Long training - learning rate decreases almost to zero after 30k steps for SmolVLA

[Grigorij-Dudnik]

System Info

Default number of training steps for SmolVLA is 20k, but sometimes we want to train it longer, eg. in case of bigger datasets. Hovewer, learning rate curve decreases at same rate independently of nr of steps choosen, and model stops to learn after 30k steps.

Information

• One of the scripts in the examples/ folder of LeRobot
• My own task or dataset (give details below)

Reproduction

I trained SmolVLA with parameter of steps set to 120000. Here is my learning rate curve:

Expected behavior

Can't say what expected behaviour should be. Maybe lr decrease rate should be proportional to nr of steps?

@jadechoghari

[jadechoghari]

@Grigorij-Dudnik can you share your training command?

[Grigorij-Dudnik]

Sure:

lerobot-train --policy.path=lerobot/smolvla_base --dataset.repo_id=Grigorij/right-arm-grab-notebook-sum --batch_size=64 --steps=120000 --policy.repo_id=smolvla_right_arm_grab_notebook --wandb.enable=true --save_checkpoint true --save_freq 20000 --rename_map='{"observation.images.main": "observation.images.camera1", "observation.images.right_arm": "observation.images.camera2"}'