application_process metrics not appearing in OBI for Node.js and Rust applications

[DarkSorrow]

Problem

OBI (OpenTelemetry eBPF Instrumentation) is successfully discovering and instrumenting our applications (Node.js/Haraka and Rust/WebMessage), but application_process metrics are not appearing in VictoriaMetrics, even though they are enabled in the configuration.

Environment

• OBI Image: otel/ebpf-instrument:main
• Deployment: DaemonSet with unprivileged mode (specific Linux capabilities)
• Backend: VictoriaMetrics (OTLP endpoint)
• Applications:
  • Haraka (Node.js) - /usr/local/bin/node
    • Base image: node:24-alpine3.20
  • WebMessage (Rust) - /usr/local/bin/webmessage
    • Base image: gcr.io/distroless/cc-debian12:nonroot (distroless container on Debian 12)

Configuration

ConfigMap (obi-config.yml)

discovery:
  instrument:
    - k8s_namespace: "mail"

otel_metrics_export:
  endpoint: victoriametrics endpoint
  protocol: http/protobuf
  compression: gzip
  encoding: proto
  tls:
    insecure: false
  features: ['application','network', 'application_process', 'application_span', 'application_host']
  buckets:
    duration_histogram: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0]
  histogram_aggregation: base2_exponential_bucket_histogram

attributes:
  kubernetes:
    enable: true
  select:
    '*':
      include: ["*"]
      exclude:
        - url.path
        - http.user_agent
        - client.address
        - server.address
  extra_group_attributes:
    k8s_app_meta: ['k8s.deployment.name']

routes:
  ignored_patterns:
    - /health
    - /health/*
    - /metrics
  ignore_mode: traces  # Only discard traces for ignored patterns, keep metrics
  unmatched: heuristic

DaemonSet Environment Variables

env:
  - name: OTEL_EBPF_KUBE_METADATA_ENABLE
    value: 'autodetect'
  - name: OTEL_EBPF_CONFIG_PATH
    value: '/config/obi-config.yml'
  - name: OTEL_EBPF_KUBE_CLUSTER_NAME
    value: 'ovh-k8s'
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: victoriametrics-auth
        key: headers
  - name: OTEL_LOG_LEVEL
    value: 'debug'
  - name: OTEL_EBPF_LOG_LEVEL
    value: 'DEBUG'

Security Context

securityContext:
  runAsUser: 0
  readOnlyRootFilesystem: true
  capabilities:
    add:
      - BPF
      - SYS_PTRACE
      - NET_RAW
      - CHECKPOINT_RESTORE
      - DAC_READ_SEARCH
      - PERFMON
      - SYS_ADMIN
    drop:
      - ALL

What's Working

✅ Process Discovery: OBI successfully discovers and instruments both applications:

time=2025-11-09T12:42:50.609Z level=DEBUG msg="found process" component=discover.CriteriaMatcher pid=272153 comm=/usr/local/bin/node metadata="map[k8s_container_name:haraka k8s_deployment_name:haraka k8s_namespace:mail k8s_pod_name:haraka-5b4bc5846-qxgmr]"

time=2025-11-09T12:42:50.729Z level=DEBUG msg=instrumented component=discover.ExecTyper pid=272129 comm=/usr/local/bin/node language=nodejs

time=2025-11-09T12:42:50.731Z level=DEBUG msg="found an instrumentable process" component=discover.ExecTyper type=rust exec=/usr/local/bin/webmessage pid=265724

✅ Traces: Traces are being submitted successfully:

time=2025-11-09T12:45:10.577Z level=DEBUG msg="submitting traces on timeout" component=ringbuf.Tracer len=1

✅ Network Metrics: OBI network metrics are being exported and received in VictoriaMetrics:

• obi.network.flow.bytes - Network flow metrics are appearing
• Network metrics are successfully exported via OTLP
• VictoriaMetrics is receiving and storing network metrics correctly

✅ HTTP Traffic: Applications receive regular HTTP traffic (health checks, etc.)

What's Not Working

❌ application_process metrics: Not appearing in VictoriaMetrics

• Query: application_process*{k8s.namespace.name="mail"} returns no results
• Query: process_cpu_utilization_ratio{k8s.namespace.name="mail"} returns no results
• Query: process_memory_usage_bytes{k8s.namespace.name="mail"} returns no results

❌ No metric export logs: We don't see "storing new metric" or metric export activity in OBI logs, even though:

• application_process is in the features list
• HTTP traffic is occurring
• Traces are being exported

Logs

Successful Instrumentation

time=2025-11-09T12:42:50.729Z level=DEBUG msg=instrumented component=discover.ExecTyper pid=272129 comm=/usr/local/bin/node comm=/usr/local/bin/node pid=272129 child=[] language=nodejs

time=2025-11-09T12:42:50.731Z level=DEBUG msg="found an instrumentable process" component=discover.ExecTyper type=rust exec=/usr/local/bin/webmessage pid=265724

Trace Submission (Working)

time=2025-11-09T12:45:10.577Z level=DEBUG msg="submitting traces on timeout" component=ringbuf.Tracer len=1

No Metric Export Logs

No "storing new metric" or metric export activity appears in logs, even with OTEL_LOG_LEVEL=debug.

Questions

1. Do application_process metrics require additional configuration? We have application_process in the features list, but is there something else needed?

2. Are application_process metrics generated differently than HTTP metrics? We see traces being exported, but no metric export logs. Should we see metric export logs at debug level?

3. Do we need to enable something else for process-level metrics (CPU, memory, disk I/O)? The application_process feature is enabled, but metrics aren't appearing.

Expected Behavior

We expect to see:

• process_cpu_utilization_ratio or similar CPU metrics
• process_memory_usage_bytes or similar memory metrics
• process_disk_io_bytes or similar disk I/O metrics

All with Kubernetes labels like k8s.namespace.name="mail", k8s.deployment.name="haraka", etc.

Additional Information

• Kubernetes Version: OVH Managed Kubernetes (latest)
• Node OS: Linux (Ubuntu-based)
• Container Runtime: containerd
• Application Container Images:
  • Haraka: node:24-alpine3.20 (Alpine Linux)
  • WebMessage: gcr.io/distroless/cc-debian12:nonroot (Distroless Debian 12)
• Network: Applications receive regular HTTP traffic
• VictoriaMetrics: Successfully receiving traces via OTLP

Related Documentation

• OBI Configuration
• OBI Metrics
• OBI Service Discovery

---

Note: We've verified that:

• Processes are being discovered and instrumented correctly
• Traces are being exported successfully
• HTTP traffic is occurring
• application_process is in the features list
• No errors appear in OBI logs

Any guidance on how to get application_process metrics working would be greatly appreciated!

[grcevski]

Hi @DarkSorrow, this is a documentation bug it seems. The application_process metrics were not part of the donation of Beyla to OTel, because of the overlap with the node exporter and potentially the OpenTelemetry collector. We had removed the exported process metrics from the list, but we forgot to remove the application_process from the options list.

[DarkSorrow]

@grcevski thanks for the reply i admit i wasted a lot of hours on it thinking i was messing up something this week end xD

For the sql part would there need to be some special implementation to also take into account database like cassandra and for redis would valkey work as well somehow? It's fine if you don't know i will try it anyway but i admit if its sure it doesn't work at least i would search a bit less :P

[grcevski]

For the sql part would there need to be some special implementation to also take into account database like cassandra and for redis would valkey work as well somehow? It's fine if you don't know i will try it anyway but i admit if its sure it doesn't work at least i would search a bit less :P

Sorry I must've missed this question :).

Cassandra might work, but we haven't tested with it. We do direct protocol detection for MySQL and Postgres, but you can enable the heuristic SQL protocol detection, which will catch any SQL like exchange. To do so you can use the config file or environment variables:

ebpf:
  heuristic_sql_detect: true

or OTEL_EBPF_HEURISTIC_SQL_DETECT=true

KeyVal should work, as far as I know the protocol is identical to Redis, we'll just recognise it as a Redis database.