[v0.3.1] GPU isolation ignored on NodeSet `esc8000a`: container sees 8 GPUs even with `nvidia.com/gpu: 1` and `GRES=gpu:a40:1`

[tsungjung411]

Summary

On NodeSet esc8000a we set both Kubernetes GPU limits (nvidia.com/gpu: 1) and Slurm GRES (gpu:a40:1). However, when running an srun on esc8000a-*, the container sees all 8 A40 GPUs via nvidia-smi. Slurm also reports Gres=gpu:a40:1 for those nodes, so Slurm’s view (1 GPU) and the container’s view (8 GPUs) are inconsistent.

We’d like guidance on the expected behavior and how to ensure that jobs (and/or the slurmd container) only see the number of GPUs granted by K8s + GRES.

---

Expected behavior

• A container scheduled with resources.limits.nvidia.com/gpu: 1 on NodeSet esc8000a should expose only one GPU to user processes (e.g., nvidia-smi lists a single device).
• srun on esc8000a-* should inherit that isolation.

Actual behavior

• nvidia-smi inside the job shows 8 GPUs on esc8000a-*.
• Slurm node state shows Gres=gpu:a40:1 for esc8000a-[0-1] (so Slurm thinks 1 GPU), but the container can access all 8.

---

Config (values excerpt)

# Base: values-cluster-v0.3.1.yaml

compute:
  nodesets:
    - name: esc8000a
      enabled: true
      replicas: 2
      nodeSelector:
        kubernetes.io/os: linux
        nvidia.com/gpu.present: "true"
        nvidia.com/gpu.machine: ESC8000-E11
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
      resources:
        limits:
          cpu: 12
          memory: 48Gi
          nvidia.com/gpu: 1
        requests:
          cpu: 8
          memory: 32Gi
          nvidia.com/gpu: 1
      useResourceLimits: true
      extraVolumeMounts:
        - name: cache
          mountPath: /dev/shm
        - name: demosite-storage
          mountPath: /mnt/data
      extraVolumes:
        - name: cache
          emptyDir:
            medium: Memory
            sizeLimit: 12Gi
        - name: demosite-storage
          persistentVolumeClaim:
            claimName: slurm-workspace-pvc
      partition:
        enabled: false
        config:
          State: UP
          MaxTime: UNLIMITED
      nodeConfig:
        Features: ["a40"]
        Gres: ["gpu:a40:1"]
        CPUs: 12
        RealMemory: 245760

---

Diagnostics

sinfo:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
tn*          up   infinite      3   idle esc8000a-[0-1],esc8000b-0

scontrol show nodes (esc8000a):

NodeName=esc8000a-0 ... 
   Gres=gpu:a40:1(S:0,2,4,6,8,10)
   Partitions=tn
   CfgTRES=cpu=11,mem=240G,billing=11,gres/gpu=1
...
NodeName=esc8000a-1 ...
   Gres=gpu:a40:1(S:0,2,4,6,8,10)
   Partitions=tn
   CfgTRES=cpu=11,mem=240G,billing=11,gres/gpu=1

scontrol show partitions (excerpt):

PartitionName=tn
   NodeSets=esc8000a,esc8000b
   Nodes=esc8000a-[0-1],esc8000b-0
   TRES=cpu=102,mem=720G,node=3,billing=102,gres/gpu=10

nvidia-smi from srun --nodelist=esc8000a-0 bash -c "hostname; nvidia-smi" (excerpt):

esc8000a-0
Driver Version: 550.54.14, CUDA 12.4
... 8x NVIDIA A40 listed ...

---

Environment

• Chart/base values: values-cluster-v0.3.1.yaml
• Slurm version (from nodes): 25.05.2
• GPU: NVIDIA A40 (8 per host on esc8000a)
• NVIDIA driver: 550.54.14
• CUDA shown: 12.4
• Kernel (esc8000a): 5.15.0-142-generic
• Partition: tn
• NodeSets: esc8000a (expected 1 GPU per pod), esc8000b (has 8 GPUs)

---

Questions / help requested

1. Should the slurmd container (and job steps launched via srun) only see the GPUs granted by resources.limits.nvidia.com/gpu on the NodeSet?
2. Does the chart expect/use a runtimeClassName: nvidia or similar to ensure the NVIDIA device plugin injects NVIDIA_VISIBLE_DEVICES for the slurmd and job containers?
3. Is there a known gap where the slurmd container runs with broader /dev/nvidia access* than the requested limit, causing jobs launched inside it to see all devices?
4. Any guidance on aligning K8s GPU limits ↔ Slurm GRES so that container device visibility matches Slurm allocation?

---

What we can provide to assist (tell us which would help)

• kubectl -n slurm get pod <slurm-compute-esc8000a-*> -o yaml (resources, runtimeClass, env)
• Env from inside slurmd container: env | grep NVIDIA, ls -l /dev/nvidia*
• Slurm config fragments resolved at runtime: slurm.conf, gres.conf as rendered in the pod
• Versions of NVIDIA GPU Operator / device plugin / container toolkit in the cluster

---

Hypotheses (for triage)

• runtimeClassName (or NVIDIA injection) not applied to slurmd / job containers → container sees all host GPUs.
• The chart’s useResourceLimits: true may not propagate nvidia.com/gpu in a way that the NVIDIA device plugin constrains device visibility.
• A mismatch between GRES and K8s device plugin (e.g., job steps executed inside the slurmd container rather than a per-job pod with GPU limits).

---

Ask
Please confirm the intended behavior and suggest the correct chart values or configuration to ensure that only 1 GPU is exposed on esc8000a-* when we set nvidia.com/gpu: 1 and Gres: ["gpu:a40:1"]. If there’s a recommended example values.yaml for 1-GPU NodeSets, that would be greatly appreciated.

[tsungjung411]

Deploy Slurm Cluster

Files

[File] run_deploy_cluster.sh

#!/bin/bash
SLURM_NAME=slurm
SLURM_VERSION=0.3.1
CHART_VALUES=values-v${SLURM_VERSION}.yaml
# curl -L https://raw.githubusercontent.com/SlinkyProject/slurm-operator/refs/tags/v${SLURM_VERSION}/helm/slurm/values.yaml -o ${CHART_VALUES}
CUSTOM_CLUSTER_VALUES=values-custom-cluster-0903.yaml
CUSTOM_COMPUTE_VALUES=values-custom-compute-0903.yaml



# [INSTALL]
echo ">>> Install ${SLURM_NAME}:"
kubectl create namespace ${SLURM_NAME}
kubectl -n ${SLURM_NAME} apply -f pvc.yaml
kubectl -n ${SLURM_NAME} apply -f regcred-ocis.yaml

helm upgrade -i "${SLURM_NAME}" oci://ghcr.io/slinkyproject/charts/slurm \
  --namespace="${SLURM_NAME}" \
  --version="${SLURM_VERSION}" \
  --values="${CHART_VALUES}" \
  -f "${CUSTOM_CLUSTER_VALUES}" \
  -f "${CUSTOM_COMPUTE_VALUES}"

echo "<<< Install: Done"

---

[File] values-custom-cluster-0903.yaml

# Base: values-cluster-v0.3.1.yaml

debug:
  #
  # -- (bool)
  # Allow a locally running operator to communicate with slurm cluster via port-forward.
  # NOTE: use when running the operator in a local debugger.
  localOperator: false

#
# -- (list)
# Set the secrets for image pull.
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
imagePullSecrets:
  - name: regcred-ocis

#
# Slurm configurations.
slurm:
  configFiles:
    # Ref: https://slurm.schedmd.com/cgroup.conf.html
    cgroup.conf: |
      CgroupPlugin=autodetect
      IgnoreSystemd=yes
      EnableControllers=yes
      ConstrainCores=yes
      ConstrainRAMSpace=yes
      ConstrainDevices=yes
      ConstrainSwapSpace=yes
    gres.conf: |
      # Ref: https://slurm.schedmd.com/gres.conf.html
      AutoDetect=nvidia
      #AutoDetect=nvml
      # no-effect #NodeName=esc8000a* Name=gpu AutoDetect=nvidia
      # no-effect #NodeName=esc8000b* Name=gpu AutoDetect=nvidia
      NodeName=e900-0 Name=gpu Type=rtx3090 File=/dev/nvidia0
      NodeName=esc8000a-[0-15] Name=gpu Type=a40
      NodeName=esc8000b-[0-15] Name=gpu Type=a40
    plugstack.conf: |
      # Ref: https://slurm.schedmd.com/plugstack.conf.html
      # Settings: https://github.com/SlinkyProject/slurm-operator/blob/main/docs/pyxis.md
      include /usr/share/pyxis/*

# Slurm authcred (sackd) configurations.
authcred:
  #
  # -- (string)
  # Set the image pull policy.
  imagePullPolicy: Always
  #
  # Set the image to use.
  image:
    #
    # -- (string)
    # Set the image tag to use.
    tag: 25.05.2-ubuntu24.04
    # See latest -> https://github.com/SlinkyProject/containers/pkgs/container/sackd

#
# Slurm controller (slurmctld) configurations.
controller:
  #
  # -- (string)
  # Set the image pull policy.
  imagePullPolicy: Always
  #
  # Set the image to use.
  image:
    #
    # -- (string)
    # Set the image tag to use.
    tag: 25.05.2-ubuntu24.04
    # See latest -> https://github.com/SlinkyProject/containers/pkgs/container/slurmctld
    #
    # -- (string)
    # Create a `PersistentVolumeClaim` with this storage class.
    storageClass: default-local
    #onDelete: Delete
  #
  # Define a persistent volume for the slurm controller to store its save-state.
  # Used to recover from system failures or from pod upgrades.
  persistence:
    #
    # -- (string)
    # Create a `PersistentVolumeClaim` with this storage class.
    storageClass: "default-local"

#
# Login node configurations.
login:
  #
  # -- (bool)
  # Enables login nodes.
  enabled: true
  #
  # -- (string)
  # Set the image pull policy.
  imagePullPolicy: Always
  #
  # Set the image to use.
  image:
    #
    # -- (string)
    # Set the image repository to use.
    repository: ghcr.io/slinkyproject/login-pyxis
    #
    # -- (string)
    # Set the image tag to use.
    tag: 25.05.2-ubuntu24.04
    # See latest -> https://github.com/SlinkyProject/containers/pkgs/container/login-pyxis
  #
  # -- (list)
  # The `/root/.ssh/authorized_keys` file to write, represented as a list.
  rootSshAuthorizedKeys:
  - ssh-rsa AAAA**************************************************
  #
  # -- (object)
  # The security context given to the container.
  # Ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
  # Settings: https://github.com/SlinkyProject/slurm-operator/blob/main/docs/pyxis.md
  securityContext:
    privileged: true
    #capabilities:
    #  add:
    #    #- SYS_CHROOT
    #    - SYS_ADMIN
    #    - SYS_PTRACE
  #
  # --(list)
  # List of volume mounts.
  # Ref: https://kubernetes.io/docs/concepts/storage/volumes/
  extraVolumeMounts:
    # - name: nfs-home
    #   mountPath: /home
    # - name: nfs-data
    #   mountPath: /mnt/data
    - name: demosite-storage
      mountPath: /mnt/data
  #
  # --(list)
  # Define list of pod volumes.
  # Ref: https://kubernetes.io/docs/concepts/storage/volumes/
  extraVolumes:
    # - name: nfs-home
    #   nfs:
    #     server: nfs-server.example.com
    #     path: /exports/home/
    # - name: nfs-data
    #   persistentVolumeClaim:
    #     claimName: nfs-data
    - name: demosite-storage  # tj
      persistentVolumeClaim:
        claimName: slurm-workspace-pvc

#
# Slurm compute (slurmd) configurations.
compute:
  #
  # -- (string)
  # Set the image pull policy.
  imagePullPolicy: Always
  #
  # Default image for the nodeset pod (slurmd)
  # Each nodeset may override this setting.
  image:
    #
    # -- (string)
    # Set the image repository to use.
    repository: ghcr.io/slinkyproject/slurmd-pyxis
    #
    # -- (string)
    # Set the image tag to use.
    tag: 25.05.2-ubuntu24.04
    # See latest -> https://github.com/SlinkyProject/containers/pkgs/container/slurmd-pyxis
    #
    # ociscloud/slinkyproject:slurmd-pyxis-25.05-ubuntu24.04
    #repository: ociscloud/slinkyproject
    #tag: slurmd-pyxis-25.05-ubuntu24.04

#
# Slurm accounting (slurmdbd) configurations.
accounting:
  #
  # -- (string)
  # Set the image pull policy.
  imagePullPolicy: Always
  #
  # Set the image to use.
  image:
    #
    # -- (string)
    # Set the image tag to use.
    tag: 25.05.2-ubuntu24.04
    # See latest -> https://github.com/SlinkyProject/containers/pkgs/container/slurmdbd

#
# `bitnami/mariadb` subchart configurations.
# Ref: https://github.com/bitnami/charts/blob/main/bitnami/mariadb/values.yaml
mariadb:
  primary:
    # NOTE: https://slurm.schedmd.com/accounting.html#slurm-accounting-configuration-before-build
    persistence:
      storageClass: default-local
      #onDelete: Delete

#
# Slurm REST API (slurmrestd) configurations.
restapi:
  #
  # -- (string)
  # Set the image pull policy.
  imagePullPolicy: Always
  #
  # Set the image to use.
  image:
    #
    # -- (string)
    # Set the image tag to use.
    tag: 25.05.2-ubuntu24.04
    # See latest -> https://github.com/SlinkyProject/containers/pkgs/container/slurmrestd

[File] values-custom-compute-0903.yaml

# Base: values-cluster-v0.3.1.yaml

#
# Slurm compute (slurmd) configurations.
compute:
  #
  # -- (list)
  # Slurm NodeSets by object list.
  nodesets:
      #
      # -- (string)
      # Name of NodeSet. Must be unique.
    - name: esc8000a
      #
      # -- (bool)
      # Enables the NodeSet in Slurm.
      enabled: true
      #
      # -- (integer)
      # Set the number of replicas to deploy.
      replicas: 2
      image:
        repository: ""
        tag: ""
      #
      # -- (string)
      # Set the priority class to use.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
      priorityClassName: ""
      #
      # -- (map)
      # Selector which must match a node's labels for the pod to be scheduled on that node.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
      nodeSelector:
        kubernetes.io/os: linux
        nvidia.com/gpu.present: "true"
        nvidia.com/gpu.machine: ESC8000-E11
        # ESC8000-E11 (node/esc8000-01)
        # ESC8000-E11P (node/ocis-esc8000-e11p-02) -> PASS
        # WS-E900-G4_WS980T (node/ws-e900-g4-ws980t) -> PASS
      #
      # -- (object)
      # Set affinity for Kubernetes Pod scheduling.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
      affinity: {}
        # podAntiAffinity:
        #   preferredDuringSchedulingIgnoredDuringExecution:
        #     - weight: 100
        #       podAffinityTerm:
        #         topologyKey: kubernetes.io/hostname
        #         labelSelector:
        #           matchExpressions:
        #             - key: app.kubernetes.io/name
        #               operator: In
        #               values:
        #                 - slurmctld
        #                 - slurmdbd
        #                 - slurmrestd
        #                 - mariadb
      #
      # -- (list)
      # Configure pod tolerations.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
      
      #
      # -- (object)
      # Set container resource requests and limits for Kubernetes Pod scheduling.
      # Ref: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-requests-and-limits-of-pod-and-container
      resources:  # {}
        limits:
          cpu: 12 
          memory: 48Gi
          nvidia.com/gpu: 1
        requests:  # tj
          cpu: 8
          memory: 32Gi
          nvidia.com/gpu: 1
      #
      # -- (bool)
      # Enable to propagate the pod `resources.limits` into slurmd.
      useResourceLimits: true
      #
      # -- (object)
      # Set the update strategy configuration.
      updateStrategy:
        #
        # -- (string)
        # Set the update strategy type.
        # Can be either: "RollingUpdate"; "OnDelete".
        type: RollingUpdate
        #
        # -- (object)
        # Define the rolling update policy.
        # Only used when "updateStrategy.type=RollingUpdate".
        rollingUpdate:
          #
          # -- (string)
          # The maximum number of pods that can be unavailable during the update.
          # Value can be an absolute number (ex: 5) or a percentage of desired
          # pods (ex: 10%). Absolute number is calculated from percentage by
          # rounding up. This can not be 0. Defaults to 1.
          maxUnavailable: 20%
      #
      # -- (object)
      # The policy used for PVCs created from the NodeSet VolumeClaimTemplates.
      persistentVolumeClaimRetentionPolicy:
        #
        # -- (string)
        # WhenDeleted specifies what happens to PVCs created from NodeSet
        # VolumeClaimTemplates when the NodeSet is deleted. The default policy
        # of `Retain` causes PVCs to not be affected by NodeSet deletion. The
        # `Delete` policy causes those PVCs to be deleted.
        whenDeleted: Retain
        #
        # -- (string)
        # WhenScaled specifies what happens to PVCs created from NodeSet
        # VolumeClaimTemplates when the NodeSet is scaled down. The default
        # policy of `Retain` causes PVCs to not be affected by a scale-in. The
        # `Delete` policy causes the associated PVCs for any excess pods to be
        # deleted.
        whenScaled: Retain
      #
      # -- (list)
      # List of PVCs to be created from template and mounted on each NodeSet pod.
      # PVCs are given a unique identity relative to each NodeSet pod.
      # Ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#volume-claim-templates
      volumeClaimTemplates: []
        # - metadata:
        #     name: scratch
        #   spec:
        #     mountPath: /mnt/scratch
        #     storageClassName: standard
        #     accessModes:
        #       - ReadWriteOnce
        #     resources:
        #       requests:
        #         storage: 1Gi
      #
      # --(list)
      # List of volume mounts.
      # Ref: https://kubernetes.io/docs/concepts/storage/volumes/
      extraVolumeMounts:  # []  # tj
        # - name: nfs-home
        #   mountPath: /home
        # - name: nfs-data
        #   mountPath: /mnt/data
        - name: cache
          mountPath: /dev/shm
        - name: demosite-storage
          mountPath: /mnt/data
      #
      # --(list)
      # Define list of pod volumes.
      # Ref: https://kubernetes.io/docs/concepts/storage/volumes/
      extraVolumes:  # [] # tj
        # - name: nfs-home
        #   nfs:
        #     server: nfs-server.example.com
        #     path: /exports/home/
        # - name: nfs-data
        #   persistentVolumeClaim:
        #     claimName: nfs-data
        - name: cache 
          emptyDir:
            medium: Memory
            sizeLimit: 12Gi
        - name: demosite-storage  # tj
          persistentVolumeClaim:
            claimName: slurm-workspace-pvc
      #
      # -- (object)
      # Partition describes the partition created specifically for this NodeSet to be added.
      partition:
        #
        # -- (bool)
        # Enables this NodeSet's partition line to be added in Slurm.
        enabled: false 
        #
        # -- (map[string]string | map[string][]string)
        # Extra Slurm partition configuration appended onto the partition line.
        # Ref: https://slurm.schedmd.com/slurm.conf.html#lbAI
        config:
          State: UP
          MaxTime: UNLIMITED
      #
      # --(map[string]string || map[string][]string)
      # Extra Slurm node configuration appended into the --conf arguments.
      # WARNING: Values can override existing ones.
      # Ref: https://slurm.schedmd.com/slurm.conf.html#lbAE
      nodeConfig:
        Features: ["a40"]    # tj
        Gres: ["gpu:a40:1"]  # tj: gres.conf: NodeName=esc8000a-[0-15] Name=gpu Type=a40
        # Weight: 1
        CPUs: 12
        RealMemory: 245760  # tj



    ########## node:ocis-esc8000-e11p-02 ##########
    - name: esc8000b
      #
      # -- (bool)
      # Enables the NodeSet in Slurm.
      enabled: true
      #
      # -- (integer)
      # Set the number of replicas to deploy.
      replicas: 1
      image:
        repository: ""
        tag: ""
      #
      # -- (string)
      # Set the priority class to use.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
      priorityClassName: ""
      #
      # -- (map)
      # Selector which must match a node's labels for the pod to be scheduled on that node.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
      nodeSelector:
        kubernetes.io/os: linux
        nvidia.com/gpu.present: "true"
        nvidia.com/gpu.machine: ESC8000-E11P
        # ESC8000-E11 (node/esc8000-01)
        # ESC8000-E11P (node/ocis-esc8000-e11p-02) -> PASS
        # WS-E900-G4_WS980T (node/ws-e900-g4-ws980t) -> PASS
      #
      # -- (object)
      # Set affinity for Kubernetes Pod scheduling.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
      affinity: {}
        # podAntiAffinity:
        #   preferredDuringSchedulingIgnoredDuringExecution:
        #     - weight: 100
        #       podAffinityTerm:
        #         topologyKey: kubernetes.io/hostname
        #         labelSelector:
        #           matchExpressions:
        #             - key: app.kubernetes.io/name
        #               operator: In
        #               values:
        #                 - slurmctld
        #                 - slurmdbd
        #                 - slurmrestd
        #                 - mariadb
      #
      # -- (list)
      # Configure pod tolerations.
      # Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
      
      #
      # -- (object)
      # Set container resource requests and limits for Kubernetes Pod scheduling.
      # Ref: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-requests-and-limits-of-pod-and-container
      resources:  # {}
        limits:
          cpu: 80
          memory: 48Gi
          nvidia.com/gpu: 8
        requests:  # tj
          cpu: 64 
          memory: 32Gi
          nvidia.com/gpu: 8
      #
      # -- (bool)
      # Enable to propagate the pod `resources.limits` into slurmd.
      useResourceLimits: true
      #
      # -- (object)
      # Set the update strategy configuration.
      updateStrategy:
        #
        # -- (string)
        # Set the update strategy type.
        # Can be either: "RollingUpdate"; "OnDelete".
        type: RollingUpdate
        #
        # -- (object)
        # Define the rolling update policy.
        # Only used when "updateStrategy.type=RollingUpdate".
        rollingUpdate:
          #
          # -- (string)
          # The maximum number of pods that can be unavailable during the update.
          # Value can be an absolute number (ex: 5) or a percentage of desired
          # pods (ex: 10%). Absolute number is calculated from percentage by
          # rounding up. This can not be 0. Defaults to 1.
          maxUnavailable: 20%
      #
      # -- (object)
      # The policy used for PVCs created from the NodeSet VolumeClaimTemplates.
      persistentVolumeClaimRetentionPolicy:
        #
        # -- (string)
        # WhenDeleted specifies what happens to PVCs created from NodeSet
        # VolumeClaimTemplates when the NodeSet is deleted. The default policy
        # of `Retain` causes PVCs to not be affected by NodeSet deletion. The
        # `Delete` policy causes those PVCs to be deleted.
        whenDeleted: Retain
        #
        # -- (string)
        # WhenScaled specifies what happens to PVCs created from NodeSet
        # VolumeClaimTemplates when the NodeSet is scaled down. The default
        # policy of `Retain` causes PVCs to not be affected by a scale-in. The
        # `Delete` policy causes the associated PVCs for any excess pods to be
        # deleted.
        whenScaled: Retain
      #
      # -- (list)
      # List of PVCs to be created from template and mounted on each NodeSet pod.
      # PVCs are given a unique identity relative to each NodeSet pod.
      # Ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#volume-claim-templates
      volumeClaimTemplates: []
        # - metadata:
        #     name: scratch
        #   spec:
        #     mountPath: /mnt/scratch
        #     storageClassName: standard
        #     accessModes:
        #       - ReadWriteOnce
        #     resources:
        #       requests:
        #         storage: 1Gi
      #
      # --(list)
      # List of volume mounts.
      # Ref: https://kubernetes.io/docs/concepts/storage/volumes/
      extraVolumeMounts:  # []
        # - name: nfs-home
        #   mountPath: /home
        # - name: nfs-data
        #   mountPath: /mnt/data
        - name: cache
          mountPath: /dev/shm
        - name: demosite-storage
          mountPath: /mnt/data
        #- name: sys-fs      
        #  mountPath: /sys/fs
      #
      # --(list)
      # Define list of pod volumes.
      # Ref: https://kubernetes.io/docs/concepts/storage/volumes/
      extraVolumes:  # []
        # - name: nfs-home
        #   nfs:
        #     server: nfs-server.example.com
        #     path: /exports/home/
        # - name: nfs-data
        #   persistentVolumeClaim:
        #     claimName: nfs-data
        - name: cache 
          emptyDir:
            medium: Memory
            sizeLimit: 12Gi
        - name: demosite-storage  # tj
          persistentVolumeClaim:
            claimName: slurm-workspace-pvc
        #- name: sys-fs
        #  hostPath:
        #    path: /sys/fs
      #
      # -- (object)
      # Partition describes the partition created specifically for this NodeSet to be added.
      partition:
        #
        # -- (bool)
        # Enables this NodeSet's partition line to be added in Slurm.
        enabled: false 
        #
        # -- (map[string]string | map[string][]string)
        # Extra Slurm partition configuration appended onto the partition line.
        # Ref: https://slurm.schedmd.com/slurm.conf.html#lbAI
        config:
          State: UP
          MaxTime: UNLIMITED
      #
      # --(map[string]string || map[string][]string)
      # Extra Slurm node configuration appended into the --conf arguments.
      # WARNING: Values can override existing ones.
      # Ref: https://slurm.schedmd.com/slurm.conf.html#lbAE
      nodeConfig:
        Features: ["a40"]    # tj
        Gres: ["gpu:a40:8"]  # tj: NodeName=esc8000b-[0-15] Name=gpu Type=a40
        # Weight: 1
        RealMemory: 245760  # tj


  #
  # -- (list)
  # Slurm Partitions by object list.
  partitions:
      #
      # -- (string)
      # Name of Partition. Must be unique.
    - name: tn
      enabled: true
      nodesets:
        - esc8000a
        - esc8000b
      config:
        State: UP
        Default: "YES"
        MaxTime: UNLIMITED

[tsungjung411]

Status

root@slurm-login-985948dff-gth6s:~# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
tn*          up   infinite      3   idle esc8000a-[0-1],esc8000b-0



root@slurm-login-985948dff-gth6s:~# scontrol show nodes
NodeName=esc8000a-1 Arch=x86_64 CoresPerSocket=1 
   CPUAlloc=0 CPUEfctv=11 CPUTot=12 CPULoad=3.45
   AvailableFeatures=esc8000a,a40
   ActiveFeatures=esc8000a,a40
   Gres=gpu:a40:1(S:0,2,4,6,8,10)
   NodeAddr=10.244.2.129 NodeHostName=esc8000a-1 Version=25.05.2
   OS=Linux 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 
   RealMemory=245760 AllocMem=0 FreeMem=196025 Sockets=12 Boards=1
   CoreSpecCount=1 CPUSpecList=11 
   State=IDLE+DYNAMIC_NORM ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=tn 
   BootTime=2025-06-21T21:40:25 SlurmdStartTime=2025-09-05T06:34:26
   LastBusyTime=2025-09-05T06:34:20 ResumeAfterTime=None
   CfgTRES=cpu=11,mem=240G,billing=11,gres/gpu=1
   AllocTRES=
   CurrentWatts=0 AveWatts=0
   Comment={"namespace":"slurm","podName":"slurm-compute-esc8000a-1"}

NodeName=esc8000a-0 Arch=x86_64 CoresPerSocket=1 
   CPUAlloc=0 CPUEfctv=11 CPUTot=12 CPULoad=3.45
   AvailableFeatures=esc8000a,a40
   ActiveFeatures=esc8000a,a40
   Gres=gpu:a40:1(S:0,2,4,6,8,10)
   NodeAddr=10.244.2.14 NodeHostName=esc8000a-0 Version=25.05.2
   OS=Linux 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 
   RealMemory=245760 AllocMem=0 FreeMem=196025 Sockets=12 Boards=1
   CoreSpecCount=1 CPUSpecList=11 
   State=IDLE+DYNAMIC_NORM ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=tn 
   BootTime=2025-06-21T21:40:25 SlurmdStartTime=2025-09-05T06:34:26
   LastBusyTime=2025-09-05T06:34:20 ResumeAfterTime=None
   CfgTRES=cpu=11,mem=240G,billing=11,gres/gpu=1
   AllocTRES=
   CurrentWatts=0 AveWatts=0
   Comment={"namespace":"slurm","podName":"slurm-compute-esc8000a-0"}

NodeName=esc8000b-0 Arch=x86_64 CoresPerSocket=20 
   CPUAlloc=0 CPUEfctv=80 CPUTot=80 CPULoad=0.18
   AvailableFeatures=esc8000b,a40
   ActiveFeatures=esc8000b,a40
   Gres=gpu:a40:8(S:0-1)
   NodeAddr=10.244.1.17 NodeHostName=esc8000b-0 Version=25.05.2
   OS=Linux 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 
   RealMemory=245760 AllocMem=0 FreeMem=123196 Sockets=2 Boards=1
   State=IDLE+DYNAMIC_NORM ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=tn 
   BootTime=2025-06-21T21:42:32 SlurmdStartTime=2025-09-05T06:34:20
   LastBusyTime=2025-09-05T06:34:26 ResumeAfterTime=None
   CfgTRES=cpu=80,mem=240G,billing=80,gres/gpu=8
   AllocTRES=
   CurrentWatts=0 AveWatts=0
   Comment={"namespace":"slurm","podName":"slurm-compute-esc8000b-0"}



root@slurm-login-985948dff-gth6s:~# scontrol show partitions 
PartitionName=tn
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   NodeSets=esc8000a,esc8000b
   Nodes=esc8000a-[0-1],esc8000b-0
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=104 TotalNodes=3 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=102,mem=720G,node=3,billing=102,gres/gpu=10



root@slurm-login-985948dff-gth6s:~# srun --nodelist=esc8000a-0 bash -c "hostname; nvidia-smi"
esc8000a-0
Fri Sep  5 06:39:10 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A40                     On  |   00000000:16:00.0 Off |                  Off |
|  0%   27C    P8             28W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A40                     On  |   00000000:38:00.0 Off |                  Off |
|  0%   38C    P0             86W /  300W |   48360MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A40                     On  |   00000000:49:00.0 Off |                  Off |
|  0%   37C    P0             85W /  300W |   48360MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A40                     On  |   00000000:5A:00.0 Off |                  Off |
|  0%   28C    P8             29W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A40                     On  |   00000000:98:00.0 Off |                  Off |
|  0%   26C    P8             27W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A40                     On  |   00000000:B8:00.0 Off |                  Off |
|  0%   26C    P8             28W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A40                     On  |   00000000:C8:00.0 Off |                  Off |
|  0%   39C    P0             88W /  300W |   46716MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A40                     On  |   00000000:D8:00.0 Off |                  Off |
|  0%   29C    P8             33W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

[tsungjung411]

[Update] /dev/nvidia* devices appear after invoking nvidia-smi

We observed a reproducible sequence suggesting on-demand device node creation / late device injection inside the job container.

Repro (on esc8000a-0):

# 1) Before touching NVML
srun -w esc8000a-0 bash -lc 'ls -ls /dev/nvidia*'
# => Only a subset is present (e.g., nvidia-modeset, nvidia-uvm, nvidia-uvm-tools, nvidia5, nvidiactl)

# 2) Run nvidia-smi once
srun -w esc8000a-0 bash -lc 'ls -ls /dev/nvidia*; nvidia-smi'

# 3) Immediately list again
srun -w esc8000a-0 bash -lc 'ls -ls /dev/nvidia*'
# => /dev now contains nvidia0..nvidia7 (timestamps match the time nvidia-smi ran)

Full logs

root@slurm-login-985948dff-h7jg9:~# srun --nodelist=esc8000a-0 bash -c "ls -ls /dev/nvidia*"
0 crw-rw-rw- 1 root root 195, 254 Sep  5 07:49 /dev/nvidia-modeset
0 crw-rw-rw- 1 root root 505,   0 Jun 21 21:48 /dev/nvidia-uvm
0 crw-rw-rw- 1 root root 505,   1 Jun 21 21:48 /dev/nvidia-uvm-tools
0 crw-rw-rw- 1 root root 195,   5 Jun 21 21:48 /dev/nvidia5
0 crw-rw-rw- 1 root root 195, 255 Jun 21 21:48 /dev/nvidiactl


root@slurm-login-985948dff-h7jg9:~# srun --nodelist=esc8000a-0 bash -c "ls -ls /dev/nvidia*; nvidia-smi"
0 crw-rw-rw- 1 root root 195, 254 Sep  5 07:49 /dev/nvidia-modeset
0 crw-rw-rw- 1 root root 505,   0 Jun 21 21:48 /dev/nvidia-uvm
0 crw-rw-rw- 1 root root 505,   1 Jun 21 21:48 /dev/nvidia-uvm-tools
0 crw-rw-rw- 1 root root 195,   5 Jun 21 21:48 /dev/nvidia5
0 crw-rw-rw- 1 root root 195, 255 Jun 21 21:48 /dev/nvidiactl
Fri Sep  5 07:57:38 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A40                     On  |   00000000:16:00.0 Off |                  Off |
|  0%   26C    P8             28W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A40                     On  |   00000000:38:00.0 Off |                  Off |
|  0%   37C    P0             86W /  300W |   48360MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A40                     On  |   00000000:49:00.0 Off |                  Off |
|  0%   36C    P0             84W /  300W |   48360MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A40                     On  |   00000000:5A:00.0 Off |                  Off |
|  0%   26C    P8             29W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A40                     On  |   00000000:98:00.0 Off |                  Off |
|  0%   25C    P8             27W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A40                     On  |   00000000:B8:00.0 Off |                  Off |
|  0%   25C    P8             28W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A40                     On  |   00000000:C8:00.0 Off |                  Off |
|  0%   38C    P0             87W /  300W |   46716MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A40                     On  |   00000000:D8:00.0 Off |                  Off |
|  0%   27C    P8             32W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+


root@slurm-login-985948dff-h7jg9:~# srun --nodelist=esc8000a-0 bash -c "ls -ls /dev/nvidia*"
0 crw-rw-rw- 1 root root 195, 254 Sep  5 07:49 /dev/nvidia-modeset
0 crw-rw-rw- 1 root root 505,   0 Jun 21 21:48 /dev/nvidia-uvm
0 crw-rw-rw- 1 root root 505,   1 Jun 21 21:48 /dev/nvidia-uvm-tools
0 crw-rw-rw- 1 root root 195,   0 Sep  5 07:57 /dev/nvidia0
0 crw-rw-rw- 1 root root 195,   1 Sep  5 07:57 /dev/nvidia1
0 crw-rw-rw- 1 root root 195,   2 Sep  5 07:57 /dev/nvidia2
0 crw-rw-rw- 1 root root 195,   3 Sep  5 07:57 /dev/nvidia3
0 crw-rw-rw- 1 root root 195,   4 Sep  5 07:57 /dev/nvidia4
0 crw-rw-rw- 1 root root 195,   5 Jun 21 21:48 /dev/nvidia5
0 crw-rw-rw- 1 root root 195,   6 Sep  5 07:57 /dev/nvidia6
0 crw-rw-rw- 1 root root 195,   7 Sep  5 07:57 /dev/nvidia7
0 crw-rw-rw- 1 root root 195, 255 Jun 21 21:48 /dev/nvidiactl

/dev/nvidia-caps:
total 0
0 cr-------- 1 root root 508, 1 Sep  5 07:57 nvidia-cap1
0 cr--r--r-- 1 root root 508, 2 Sep  5 07:57 nvidia-cap2

[vivian-hafener]

I will look into this. Thank you for providing all of the above information.

[vivian-hafener]

Good afternoon @tsungjung411,

I believe that this may be the expected behavior of the Kubernetes device plugin for Nvidia GPUs, and may not indicate a bug. I believe that nvidia-smi will show all GPUs present on a host, even when only a single GPU is attached to the pod. To confirm that the Nvidia k8s-device-plugin works, check the NVIDIA_VISIBLE_DEVICES variable within your worker pod. This variable is used by the Nvidia Kubernetes device plugin to limit the devices available within an environment. [1] [2]

Here is an example in my environment, showing the GPUs visible to nvidia-smi, along with the GPUs that are actually available to the pod:

ponderosa at ~ ❯ kubectl exec -n slurm slurm-worker-slinky-0 -- nvidia-smi
Tue Oct  7 20:48:10 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.261.03             Driver Version: 535.261.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA L4                      Off | 00000000:00:03.0 Off |                    0 |
| N/A   49C    P8              17W /  72W |      0MiB / 23034MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA L4                      Off | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P8              16W /  72W |      0MiB / 23034MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
ponderosa at ~ ❯ kubectl exec -n slurm slurm-worker-slinky-0 -- nvidia-smi -L
GPU 0: NVIDIA L4 (UUID: GPU-a471d35e-950f-1a91-48db-0e3b578ac8ed)
GPU 1: NVIDIA L4 (UUID: GPU-a8301131-b3ca-b155-cc61-a4d72d00f0d7)

ponderosa at ~ ❯ kubectl exec -n slurm slurm-worker-slinky-0 -- printenv | grep NVIDIA
NVIDIA_VISIBLE_DEVICES=GPU-a471d35e-950f-1a91-48db-0e3b578ac8ed

When run in your environment, is the NVIDIA_VISIBLE_DEVICES variable being set properly?

Best,
Vivian Hafener

Sources:
[1] NVIDIA/k8s-device-plugin#197
[2] NVIDIA/k8s-device-plugin#236