Training Script Stuck at 0% with High GPU Utilization

[EkSulfur]

While running train.py on a specific dataset, the training process gets stuck at 0% progress. After investigating with the Python Debugger, I found that the script runs extremely slowly and hangs at the following line in mip-splatting/train.py:

ema_loss_for_log = 0.4 * loss.item() + 0.6 * ema_loss_for_log # Line 143

The GPU is consistently at full load during this time, but no progress is made. I’m unable to determine the root cause of this issue and would appreciate any guidance or suggestions.

Environment Details:

OS: WSL Ubuntu 22.04

CUDA Version: 11.8 (Cuda compilation tools, release 11.8, V11.8.89)

Thank you in advance for your help!