Bug(multiprocess): handle inconsistent PID namespace with NVML API

[Shouren]

Description

In some version of NVIDIA Driver, e.g., Driver Version: 560.35.03, when calling nvmlDeviceGetComputeRunningProcesses and nvmlDeviceGetProcessUtilization, we may get the pid from different PID namespace.

I create a gist for reproduce the issue and test it on a server with the following config

• Driver Version: 560.35.03
• CUDA Version: 12.6

run a docker container with

docker run --rm -it --name shouren-hami-core-tester1 \
	--entrypoint /bin/bash \
	--gpus '"device=3"' \
	-v /root/shouren/Project-HAMi/hami-core:/opt/hami-core \
        nvcr.io/nvidia/pytorch:24.06-py3

get the output of the test

root@23713ecf3450:/opt/hami-core/tmp# ./pid_check
main: pid: 323
main: compute thread started successfully
Found 1 NVIDIA GPU device(s).

Device 0: NVIDIA GeForce RTX 2080 Ti, GPU-ea5a7f29-c714-06ba-23ef-cedbc44248da
Num of running process : 1
    process id: 324	mem: 0
Failed to get processes utils by device with Not Found
Found 1 NVIDIA GPU device(s).

Device 0: NVIDIA GeForce RTX 2080 Ti, GPU-ea5a7f29-c714-06ba-23ef-cedbc44248da
Num of running process : 1
    process id: 324	mem: 0
Num of process sampled: 1
    process id: 645400	smUtil: 13
Found 1 NVIDIA GPU device(s).
...

As it shows the pid given by nvmlDeviceGetComputeRunningProcesses function is in container's namespace, however the pid given by nvmlDeviceGetProcessUtilization is from host pid namepace.

I think it may be the root cause for issue like #104 and some other issues in HAMi repo.

Solution

I want to know if we should handle this inconsistent behavior ?

[luohua13]

Request 10% core and consume 100% in fact, it occur in driver version 550 and 570. Is this related to the problem described above?

[almazkhalikov]

Hey! I’ve faced the same issue with NVML returning mismatched PIDs.

• nvmlDeviceGetComputeRunningProcesses() → returns container PID (e.g. 375)
• nvmlDeviceGetProcessUtilization() → returns host PID (e.g. 604901)

This makes HAMi unable to match utilization samples to processes inside containers, so userutil always ends up 0.

Environment:

• NVIDIA Driver: 560.xx
• CUDA: 12.6
• Kubernetes + container runtime (containerd)
• HAMi-core: master branch

Questions:

1. Will upgrading to a newer NVIDIA driver (or CUDA toolkit) solve this mismatch?
2. Or is this a bug in NVML that should be reported directly to NVIDIA?

[Shouren]

Questions:

1. Will upgrading to a newer NVIDIA driver (or CUDA toolkit) solve this mismatch?

@luohua13 Upgrade to a newer NVIDIA Driver will solve this problem, i had test it on Driver Version: 570.172.08 & CUDA 12.8