[Bug]: slurm controller Pod start error

[JornShen]

Description

[Thu Nov 13 07:26:17 UTC 2025] Start '/etc/slurm' polling                                                                                                                  │ [Thu Nov 13 07:26:17 UTC 2025] Reconfiguring Slurm...
│ slurm_reconfigure error:Unable to contact slurm controller (connect failure)                                                                                               │ [Thu Nov 13 07:26:27 UTC 2025] Failed to reconfigure, try again...
│ slurm_reconfigure error:Unable to contact slurm controller (connect failure)                                                                                               │ [Thu Nov 13 07:26:39 UTC 2025] Failed to reconfigure, try again...
│ slurm_reconfigure error:Unable to contact slurm controller (connect failure)                                                                                               │ [Thu Nov 13 07:26:52 UTC 2025] Failed to reconfigure, try again...
│ slurm_reconfigure error:Unable to contact slurm controller (connect failure)                                                                                               │ [Thu Nov 13 07:27:04 UTC 2025] Failed to reconfigure, try again...

Steps to Reproduce

Expected Behavior

Additional Context

[JornShen]

is it a bug ? when i run slurm.

[SkylerMalinowski]

What version of Kubernetes are you using? I suspect (given #78) that you are < 1.29.

[und-miller]

@SkylerMalinowski I am also having an issue where my slurm-controller-0 is not starting. See my logs below.

Slinky version: 1.0.0-rc1

$ kubectl version
Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.33.1

Container: slurmctld
Filter
Disconnected
2025-11-20 16:11:44,804 INFO Included extra file "/etc/supervisor/conf.d/slurmctld.conf" during parsing
2025-11-20 16:11:44,806 INFO supervisord started with pid 1
2025-11-20 16:11:45,810 INFO spawned: 'slurmctld' with pid 12
+ exec slurmctld --systemd
[2025-11-20T16:11:45] warning: _prepare_run_dir: /run/slurmctld exists but is owned by 0, not SlurmUser
[2025-11-20T16:11:45] error: Configured MailProg is invalid
[2025-11-20T16:11:45] slurmctld version 25.11.0 started on cluster a-kbs-slinky_a-kbs-slinky-slurm(3873)
[2025-11-20T16:11:45] error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:199: pmi/pmix: can not load PMIx library
[2025-11-20T16:11:45] error: Couldn't load specified plugin name for mpi/pmix: Plugin init() callback failed
[2025-11-20T16:11:45] error: MPI: Cannot create context for mpi/pmix
[2025-11-20T16:11:45] error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:199: pmi/pmix: can not load PMIx library
[2025-11-20T16:11:45] error: Couldn't load specified plugin name for mpi/pmix_v5: Plugin init() callback failed
[2025-11-20T16:11:45] error: MPI: Cannot create context for mpi/pmix_v5
[2025-11-20T16:11:45] error: xsystemd_change_mainpid: connect() failed for /tmp/fakesystemd.sock: No such file or directory
[2025-11-20T16:11:45] No memory enforcing mechanism configured.
[2025-11-20T16:11:45] Recovered state of 0 nodes
[2025-11-20T16:11:45] error: ################################################
[2025-11-20T16:11:45] error: ###       SEVERE SECURITY VULNERABILITY       ###
[2025-11-20T16:11:45] error: ### StateSaveLocation DIRECTORY IS WORLD WRITABLE ###
[2025-11-20T16:11:45] error: ###         CORRECT FILE PERMISSIONS         ###
[2025-11-20T16:11:45] error: ################################################
[2025-11-20T16:11:45] Recovered information about 0 jobs
[2025-11-20T16:11:45] build_part_bitmap: No nodes in partition slinky
[2025-11-20T16:11:45] build_part_bitmap: No nodes in partition all
[2025-11-20T16:11:45] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 2 partitions
[2025-11-20T16:11:45] Recovered state of 0 reservations
[2025-11-20T16:11:45] read_slurm_conf: backup_controller not specified
[2025-11-20T16:11:45] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure
[2025-11-20T16:11:45] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 2 partitions
[2025-11-20T16:11:45] Running as primary controller
2025-11-20 16:11:46,860 INFO success: slurmctld entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-11-20 16:12:44,711 WARN received SIGTERM indicating exit request
2025-11-20 16:12:44,711 INFO waiting for slurmctld to die
[2025-11-20T16:12:44] Terminate signal SIGTERM received
[2025-11-20T16:12:46] Saving all slurm state
2025-11-20 16:12:46,878 INFO stopped: slurmctld (exit status 0)