Enroot overlay filesystem compatibility issue in Kubernetes environments

[kotarusv]

Environment:

Slinky Operator: v0.3.1
Kubernetes
Storage: local-path storage class (overlay filesystem)
Images: ghcr.io/slinkyproject/login-pyxis:25.05-ubuntu24.04, ghcr.io/slinkyproject/slurmd-pyxis:25.05-ubuntu24.04

Issue Summary:
We successfully implemented container support using Slinky but encounter overlay filesystem errors with complex container images. Simple images work correctly, but multi-layer images fail during enroot's overlay filesystem operations.

Two Approaches Tested:

Approach 1: OCI Runtime Integration

slurm:
configFiles:
oci.conf: |
OCIRuntime=/usr/bin/enroot
EnvExclude=HOME,MAIL,USER,SHELL,LOGNAME
RunTimeCreate=/usr/bin/enroot create
RunTimeDelete=/usr/bin/enroot remove

Approach 2: Pyxis SPANK Plugin

login:
image:
repository: ghcr.io/slinkyproject/login-pyxis
compute:
image:
repository: ghcr.io/slinkyproject/slurmd-pyxis
slurm:
configFiles:
plugstack.conf: |
include /usr/share/pyxis/*

Error (identical in both approaches):
enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.*/operation not permitted

Failed Error scanario:

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=postgres:13 date
srun: unrecognized option '--container-image=postgres:13'
Try "srun --help" for more information
command terminated with exit code 255
(venv) [ubuntu@skotaru-lnx slinky (⎈ |kai-pdc-oidc:slurm)]$ kubectl describe pod -n slurm slurm-login-b666757b8-7bv9v | grep -i image
Image: ghcr.io/slinkyproject/sackd:25.05-ubuntu24.04
Image ID: ghcr.io/slinkyproject/sackd@sha256:fe75328f91b22600261e5b65fa877830703608e9ea38eb3454ccaf28ed8407fb
Image: ghcr.io/slinkyproject/login:25.05-ubuntu24.04
Image ID: ghcr.io/slinkyproject/login@sha256:938c5abe666325ca00525fe9efe2209000638a703be717cd3f4b50882bb28fc8
Normal Pulled 66s kubelet Container image "ghcr.io/slinkyproject/sackd:25.05-ubuntu24.04" already present on machine
Normal Pulled 66s kubelet Container image "ghcr.io/slinkyproject/login:25.05-ubuntu24.04" already present on machine
(venv) [ubuntu@skotaru-lnx slinky (⎈ |kai-pdc-oidc:slurm)]$ kubectl describe pod -n slurm slurm-compute-debug-0 | grep -i image
Image: ghcr.io/slinkyproject/sackd:25.05-ubuntu24.04
Image ID: ghcr.io/slinkyproject/sackd@sha256:fe75328f91b22600261e5b65fa877830703608e9ea38eb3454ccaf28ed8407fb
Image: ghcr.io/slinkyproject/sackd:25.05-ubuntu24.04
Image ID: ghcr.io/slinkyproject/sackd@sha256:fe75328f91b22600261e5b65fa877830703608e9ea38eb3454ccaf28ed8407fb
Image: ghcr.io/slinkyproject/slurmd:25.05-ubuntu24.04
Image ID: ghcr.io/slinkyproject/slurmd@sha256:759a0573d18597ed39dcc41e6b6c6060a85d72cdd828ccf4217a13c57717d002
Normal Pulled 65s kubelet Container image "ghcr.io/slinkyproject/sackd:25.05-ubuntu24.04" already present on machine
Normal Pulled 64s kubelet Container image "ghcr.io/slinkyproject/sackd:25.05-ubuntu24.04" already present on machine
Normal Pulled 63s kubelet Container image "ghcr.io/slinkyproject/slurmd:25.05-ubuntu24.04" already present on machine

Successful Scanario:

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=alpine date
pyxis: importing docker image: alpine
pyxis: imported docker image: alpine
Wed Sep 3 21:35:37 UTC 2025

Working Examples:
alpine:latest - Success
python:3.9 - Success
ubuntu:20.04 - Success

Failing Examples:
postgres:13 - Overlay filesystem error
nvcr.io#nvidia/pytorch:23.10-py3 - Overlay filesystem error

Questions:
Is enroot/pyxis officially supported in Kubernetes environments with overlay storage?
Are there recommended configurations for avoiding overlay-on-overlay filesystem conflicts?
Should we expect limitations with complex multi-layer container images in this environment?

Additional Context:
The underlying issue appears to be enroot attempting to create overlay filesystems on top of Kubernetes' existing overlay storage, resulting in nested overlay operations that fail with permission errors.

Srinivas Kotaru

[kotarusv]

Can someone take a look and guide us ?

[vivian-hafener]

The logs you have provided do not provide sufficient context for me to understand the "overlay filesystem error" you mention.

The logs in your "Failed Error scanario" show you attempting to run srun with --container-image= in a non-pyxis image. --container-image is not a supported command-line argument for srun except when running with pyxis. Please see the Pyxis documentation and the documentation for srun for more information on supported command-line arguments.

[kotarusv]

I configured pyxis using below document .
https://github.com/SlinkyProject/slurm-operator/blob/release-0.3/docs/pyxis.md

cat pyxis-plugstack-only.yaml
login:
image:
repository: ghcr.io/slinkyproject/login-pyxis
tag: 25.05-ubuntu24.04
securityContext:
privileged: true

compute:
image:
repository: ghcr.io/slinkyproject/slurmd-pyxis
tag: 25.05-ubuntu24.04

slurm:
configFiles:
plugstack.conf: |
include /usr/share/pyxis/*

it is working for some images like below

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=python:3.9 bash -c "python -c 'import sys; print(sys.version)'"
pyxis: importing docker image: python:3.9
pyxis: imported docker image: python:3.9
3.9.23 (main, Aug 12 2025, 23:06:01)
[GCC 14.2.0]

It failing some images like below error

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=postgres:13 date
pyxis: importing docker image: postgres:13
[2025-09-03T20:15:44.818] error: pyxis: child 22127 failed with error code: 2
[2025-09-03T20:15:44.818] error: pyxis: failed to import docker image
[2025-09-03T20:15:44.818] error: pyxis: printing enroot log file:
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Querying registry for permission grant
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Authenticating with user:
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Authentication succeeded
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Fetching image manifest list
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Fetching image manifest
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Downloading 14 missing layers...
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Extracting image layers...
[2025-09-03T20:15:44.818] error: pyxis: [INFO] Converting whiteouts...
[2025-09-03T20:15:44.818] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.sLzEcMcpqH/10/var/lib/apt/lists/auxfiles: Operation not permitted
[2025-09-03T20:15:44.818] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.sLzEcMcpqH/12/etc/alternatives/pager.1.gz: Operation not permitted
[2025-09-03T20:15:44.818] error: pyxis: couldn't start container
[2025-09-03T20:15:44.819] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2025-09-03T20:15:44.819] error: Failed to invoke spank plugin stack
[2025-09-03T20:15:44.835] error: pyxis: child 22726 failed with error code: 1
srun: error: debug-1: task 0: Exited with exit code 1
command terminated with exit code 1

ask me which additional config if you want to take a look

[kotarusv]

My intention is to run this image..

nvcr.io#nvidia/pytorch:21.12-py3

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=nvcr.io#nvidia/pytorch:23.10-py3 python --version
pyxis: importing docker image: nvcr.io#nvidia/pytorch:23.10-py3
[2025-09-03T20:08:31.782] error: pyxis: child 18011 failed with error code: 7
[2025-09-03T20:08:31.782] error: pyxis: failed to import docker image
[2025-09-03T20:08:31.782] error: pyxis: printing enroot log file:
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Querying registry for permission grant
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Authenticating with user:
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Authentication succeeded
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Fetching image manifest list
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Fetching image manifest
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Found all layers in cache
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Extracting image layers...
[2025-09-03T20:08:31.782] error: pyxis: [INFO] Converting whiteouts...
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/17/usr/local/lib/python3.10/dist-packages/PIL/QoiImagePlugin.py: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/15/usr/local/lib/python3.10/dist-packages/charset_normalizer/main.py: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/18/usr/local/lib/python3.10/dist-packages/google/protobuf/internal/_api_implementation.cpython-310-x86_64-linux-gnu.so: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/28/usr/local/share/jupyter/lab/static/main.7c8bc320e7b7d81124c1.js: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/31/usr/local/lib/libtbb.so: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/40/tmp/cuda-9.0.patch: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.W8Wwh4md4c/50/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease: Operation not permitted
[2025-09-03T20:08:31.782] error: pyxis: couldn't start container
[2025-09-03T20:08:31.782] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2025-09-03T20:08:31.782] error: Failed to invoke spank plugin stack
[2025-09-03T20:08:31.798] error: pyxis: child 18626 failed with error code: 1
srun: error: debug-1: task 0: Exited with exit code 1
command terminated with exit code 1

[vivian-hafener]

Good morning @kotarusv,

It has been brought to my attention that it is a known limitation of enroot that the tmp directory often runs into issues on older NFS or Lustre filesystems due to limitations with those implementations. Please see attached resources for more information. Do you still encounter this issue when using storage that is not based on NFS or Lustre?

Best,
Vivian Hafener

NVIDIA/pyxis#87
https://github.com/NVIDIA/enroot/blob/main/doc/requirements.md#kernel-configuration

[kotarusv]

Thanks @vivian-hafener
I am not using NFS or Lustre. In fact am not using any PVC

all debug pods are using local node fS

I also verified the test you recommended. I kubectl rsh into one debug compute pod and downloaded the enroot check script

kubectl get pods -n slurm
NAME READY STATUS RESTARTS AGE
slurm-accounting-0 1/1 Running 0 50m
slurm-compute-debug-0 2/2 Running 0 39m
slurm-compute-debug-1 2/2 Running 0 39m
slurm-compute-debug-2 2/2 Running 0 39m
slurm-controller-0 3/3 Running 0 39m
slurm-exporter-86d4cf7fdb-s8zns 1/1 Running 0 50m
slurm-login-6c66859879-6l7vr 1/1 Running 0 39m
slurm-mariadb-0 1/1 Running 0 50m
slurm-restapi-d8f575b5-2crmv 1/1 Running 0 39m

kubectl exec -it slurm-compute-debug-0 -- bash

root@debug-1:/tmp# curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v3.5.0/enroot-check_3.5.0_$(uname -m).run
root@debug-1:/tmp# chmod +x enroot-check_3.5.0_x86_64.run
root@debug-1:/tmp# ./enroot-check_3.5.0_x86_64.run
Extracting [####################] 100%
Bundle ran successfully!

root@debug-0:/tmp# df -PH .
Filesystem Size Used Avail Use% Mounted on
overlay 1.9T 53G 1.8T 3% /

Do you think am still missing something?

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=postgres:13 date
pyxis: importing docker image: postgres:13
[2025-09-08T19:32:13.744] error: pyxis: child 5407 failed with error code: 2
[2025-09-08T19:32:13.744] error: pyxis: failed to import docker image
[2025-09-08T19:32:13.744] error: pyxis: printing enroot log file:
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Querying registry for permission grant
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Authenticating with user:
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Authentication succeeded
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Fetching image manifest list
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Fetching image manifest
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Found all layers in cache
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Extracting image layers...
[2025-09-08T19:32:13.744] error: pyxis: [INFO] Converting whiteouts...
[2025-09-08T19:32:13.744] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.1AK2ZGMCZC/10/var/lib/apt/lists/auxfiles: Operation not permitted
[2025-09-08T19:32:13.744] error: pyxis: enroot-aufs2ovlfs: failed to create ovlfs whiteout: /tmp/enroot.1AK2ZGMCZC/12/etc/alternatives/pager.1.gz: Operation not permitted
[2025-09-08T19:32:13.744] error: pyxis: couldn't start container
[2025-09-08T19:32:13.744] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2025-09-08T19:32:13.744] error: Failed to invoke spank plugin stack
[2025-09-08T19:32:13.764] error: pyxis: child 5607 failed with error code: 1
srun: error: debug-2: task 0: Exited with exit code 1

I am still seeing the same whiteout issue ..

As I said, it is working for some simple images

kubectl exec -n slurm deployment/slurm-login -- srun --partition=debug --container-image=alpine:latest echo "Container test successful"
pyxis: importing docker image: alpine:latest
pyxis: imported docker image: alpine:latest
Container test successful

Please help or advise what we are missing here??

Srinivas Kotaru

[kotarusv]

Specific error:
enroot-aufs2ovlfs: failed to create ovlfs whiteout: Operation not permitted

[vivian-hafener]

Good afternoon @kotarusv ,

This looks more involved than I initially expected. Please file a ticket with https://support.schedmd.com/. Official support for Slinky and Slurm is handled from that site, as the development and support teams have different priorities.

Best,
Vivian Hafener

[kotarusv]

Also keep in mind, I am using K8s with Slinky, so let us troubleshoot this way rather than applying traditional HPC troubleshooting to a containerized environment.

[kotarusv]

Key Distinction:

Working: Simple images (alpine, python, ubuntu)
Failing: Complex multi-layer images during "Converting whiteouts" phase
Environment: Kubernetes pods with overlay filesystem (not NFS/Lustre)

Request:
Could you provide guidance specific to Kubernetes containerized deployments where enroot operates within overlay filesystem environments? The traditional VM-based troubleshooting steps don't apply to this containerized use case.
The issue appears to be overlay-on-overlay filesystem limitations rather than the NFS/Lustre constraints referenced in your response.

[vivian-hafener]

Hi @kotarusv , does SlinkyProject/containers@efc20ba resolve your issue?