dotnet monitor 8 and 9 throwing error Unable to discover a target process with status 400

[Surathkal]

I'm trying to set up dotnet monitor (8 or 9) to get GC and other details as per the example https://github.com/dotnet/dotnet-monitor/blob/main/samples/Kubernetes/deployment.yaml
However, when I try to hit any HTTP end point like /process or /dump?pid=1 I get the following response

I would greatly appreciate it, if someone could please help on what exactly am I missing here ?

{"status":400,"detail":"Unable to discover a target process."}

My deployment yaml is as follows.

apiVersion: apps/v1
kind: Deployment
metadata:
name: upload-nag-test
namespace: csx
labels:
app: upload-nag-test
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app: upload-nag-test
template:
metadata:
labels:
app: upload-nag-test
azure.workload.identity/use: "true"
spec:
serviceAccountName: workload-identity-sa-csx

  containers:

  - name: upload-nag-test
    image: kmmp.azurecr.io/csx/upload:IMAGE_TAG
    imagePullPolicy: Always
    resources:
      requests:
        memory: "800Mi"
        cpu: "100m"
      limits:
        memory: "800Mi"
    ports:
      - containerPort: 8080

    envFrom:
      - configMapRef:
          name: upload-nag-test-config-map

    env:
    - name: DOTNET_DiagnosticPorts
      value: "/diag/dotnet-monitor.sock"

    volumeMounts:

    - name: secrets-store01-inline
      mountPath: /usr/share/secrets-store
      readOnly: true

    - name: diagvol
      mountPath: /diag

  - name: monitor
    image: mcr.microsoft.com/dotnet/monitor:9

    args: [ "collect", "--no-auth" ]

    imagePullPolicy: Always

    env:

    - name: DOTNETMONITOR_Urls
      value: http://localhost:52323

    - name: DOTNETMONITOR_Metrics__Endpoints
      value: http://+:52325

    - name: DotnetMonitor_DiagnosticPort__ConnectionMode
      value: "Listen"

    - name: DOTNETMONITOR_Storage__DefaultSharedPath
      value: "/diag"


    volumeMounts:

    - name: diagvol
      mountPath: /diag

    resources:
      requests:
        cpu: 50m
        memory: 32Mi
      limits:
        cpu: 250m
        memory: 256Mi

  volumes:
  
  - name: secrets-store01-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-kv-workload-identity-upload-nag-test"

  - name: diagvol
    emptyDir: {}

When I try to view the logs from the dotnet monitor container (with the following), everything looks good except when HTTP end point like /process or /dump is run I see the following exception in the code.
kubectl logs pod/upload-nag-test-bd5744b9c-l6xlf -n csx -c monitor

{
"Timestamp": "2025-05-16T15:14:41.6468959Z",
"EventId": 1,
"LogLevel": "Error",
"Category": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController",
"Message": "Request failed.",
"Exception": "System.ArgumentException: Unable to discover a target process.\n at Microsoft.Diagnostics.Monitoring.WebApi.DiagnosticServices.GetProcessAsync(DiagProcessFilter processFilterConfig, CancellationToken token)\n at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagnosticsControllerBase.\u003C\u003Ec__DisplayClass5_0\u00601.\u003C\u003CInvokeForProcess\u003Eb__0\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.ControllerExtensions.InvokeService[T](ControllerBase controller, Func\u00601 serviceCall, ILogger logger)",
"State": {
"Message": "Request failed.",
"{OriginalFormat}": "Request failed."
},
"Scopes": [
{
"Message": "SpanId:47a5fc8975c6f6b5, TraceId:db65faaaf87ba856a59f900c6d3ec988, ParentId:0000000000000000",
"SpanId": "47a5fc8975c6f6b5",
"TraceId": "db65faaaf87ba856a59f900c6d3ec988",
"ParentId": "0000000000000000"
},
{
"Message": "ConnectionId:0HNCKHHGPMOGO",
"ConnectionId": "0HNCKHHGPMOGO"
},
{
"Message": "RequestPath:/process RequestId:0HNCKHHGPMOGO:00000001",
"RequestId": "0HNCKHHGPMOGO:00000001",
"RequestPath": "/process"
},
{
"Message": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.GetProcessInfo (Microsoft.Diagnostics.Monitoring.WebApi)",
"ActionId": "dc99c1de-9b04-44f0-9682-08754d329d4e",
"ActionName": "Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.GetProcessInfo (Microsoft.Diagnostics.Monitoring.WebApi)"
}
]
}

[Josefczak]

We have the same situation and error. With the same setup it sometimes works and sometimes not.