Slurm scheduler cant be access via ssh or bastion

Hi Jason Gumarang,

What changes were altered in the setup and were any network settings been changed.

Could you please confirm if any changes were performed in configuration files or scripts that manage the scheduler.

Hi Support,

Sorry for the late response.

Basically, there was no changes made on the scheduler, it seems some of my colleague accidentally run their job locally in the scheduler (not using the slurm), causing the scheduler to be unstable due to low resource.

So, there was no setting that were altered or in the network settings, no changes as well in the configuration files or the scripts. Is just these two observations i cant find a way to retain its public network IP (Terminate) and its /share/home folder after reboot. When i reboot, the public IP can remain but the /share/home folder does not retain (causing my users unable to login because the home folder is blank, no ssh keys)

regards,
Jason

Hello Jason Gumarang, it appears that there is a resource exhaustion on the scheduler node most likely due to someone running a heavy job directly on the scheduler VM instead of submitting through Slurm. Could you verify it once from your end?

and to answer your question- If I restart the scheduler, will current jobs on compute nodes be cancelled?
Ans- No restarting the scheduler VM does not immediately cancel running jobs on compute nodes. However, the Slurm controller on the scheduler will temporarily lose communication with compute nodes. If the downtime is for long, the compute nodes may mark the jobs as NODE_FAIL. Once the scheduler comes back up and slurmctld resumes, it should attempt to reconnect with the nodes. I would suggest before restarting, gracefully stop slurmctld and bring it back up cleanly

sudo systemctl stop slurmctld

sudo reboot

and then sudo systemctl start slurmctld

your next question- Why is /share/home not mounted after reboot?
Ans- please once check if the /etc/fstab entry has noauto or is missing x-systemd.automount.

if not there then add it- <your-nfs-server>:/share/home /share/home nfs defaults,_netdev,x-systemd.automount 0 0

after that reboot and verify- mount | grep /share/home

if still not mounted then manually do a sudo mount -a

your next question- How to retain the public IP of the scheduler if I terminate or redeploy?
Ans- In Azure, a VM’s public IP is dynamic by default, meaning it changes on deallocation or termination. If you want, you can update it to static or create a new static ip and assign it to your VM's NIC

• Resource exhaustion - yes, this is correct. the scheduler ran out of resources.
• Scheduler Restart - Got it. This is clear.
• Why is /share/home not mounted after reboot? - it seems cyclecloud has a way (maybe through jetbpack) to mount my /share/home. So basically, my /home is somewhat link to /share/home. All of my users in my cyclecloud user are sync in my /share/home and the weird part is that /home and /share/home are not soft link to each other. It seems cycloud does not include a persistent configuration in my fstab but it shows in df -h-

fstab file:

UUID=b2f6e39f-3593-4dd6-a2ab-c020bcfe5189 /                       xfs     defaults        0 0

UUID=c04fc77f-2590-4b64-842b-3ded3a0df30f /boot                   xfs     defaults        0 0

UUID=2822-2755          /boot/efi               vfat    defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2

/dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,_netdev,comment=cloudconfig 0 2

df -h

Filesystem                                                    Size  Used Avail Use% Mounted on

devtmpfs                                                      7.8G     0  7.8G   0% /dev

tmpfs                                                         7.9G     0  7.9G   0% /dev/shm

tmpfs                                                         7.9G  777M  7.1G  10% /run

tmpfs                                                         7.9G     0  7.9G   0% /sys/fs/cgroup

/dev/sdc4                                                      63G   44G   20G  69% /

/dev/sdc3                                                    1014M  277M  738M  28% /boot

/dev/sdc2                                                     200M  5.9M  194M   3% /boot/efi

/dev/sdd1                                                     147G  164K  140G   1% /mnt

/dev/mapper/vg_cyclecloud_builtinsched-lv0                     30G  247M   30G   1% /sched

/dev/mapper/vg_cyclecloud_builtinshared-lv0                   2.0T  127G  1.9T   7% /shared

i tried adding my logical volume in my fstab and shutdown and start the scheduler, now, the scheduler is just stuck "waiting for virtual machine" in my cycloud dashboard.

Hello Jason Gumarang, correct, in Azure CycleCloud, the /home and /shared/home (or /share/home) relationship is not always a simple symlink. Instead, Jetpack scripts usually mount a logical volume (like /shared) and dynamically bind-mount user directories into /home. This happens during Jetpack runtime configuration, and not through /etc/fstab.

Why /home and /share/home are not in fstab?

ans- CycleCloud uses Jetpack (usually /opt/cycle/jetpack) to handle shared filesystem mounts (/shared, /sched) or user sync and SSH key injection or for mounting /shared/home/<username> into /home/<username> using bind mounts at runtime

So, the absence in your /etc/fstab is expected as it’s Jetpack’s job.

Why did the scheduler VM hang on “Waiting for virtual machine”?

Ans- This likely happened because of a misconfiguration in fstab. If you manually added a logical volume (like /shared) to fstab but didn't match the correct device path or used incorrect options, the OS might fail to boot, hang at mount time, or Jetpack services could get stuck waiting. In Azure CycleCloud, this breaks VM startup because Jetpack waits for dependencies like mount points to be ready or a failed fstab mount can block systemd from completing boot, making the VM appear “stuck”.

What is the way to Get Scheduler Back?

Ans- Start VM in Recovery Mode (or attach disk to another VM). If you can't SSH or connect via serial console, use Azure Portal to stop the scheduler VM and detach the OS disk and attach it to another Linux VM as a data disk.

Mount the OS Disk and Edit fstab-

sudo mkdir /mnt/scheduler
sudo mount /dev/sdX4 /mnt/scheduler    # Replace sdX with actual disk ID

sudo nano /mnt/scheduler/etc/fstab

Comment out or remove the custom /shared or logical volume mount you added. Save, unmount, and reattach the disk to original scheduler VM using sudo umount /mnt/scheduler

Start the scheduler VM. It should now boot normally and reappear in CycleCloud.