multi-node LLM fine-tuning

[skb888]

I have read this readme file: https://github.com/kubeflow/trainer/tree/master/docs/proposals/2401-llm-trainer-v2
It seems that TorchTuneConfig can support for multi-node LLM fine-tuning. Is this feature on the roadmap? If so, when it will be ready? It seems that currently Python SDK kubeflow V0.1.0 does not support for peft_config and multi-node, which is also highlighted in the doc: https://www.kubeflow.org/docs/components/trainer/user-guides/builtin-trainer/torchtune/ (It’s worth noticing that we do not support multi-node fine-tuning with TorchTune.) Do I have correct understanding?

[Electronic-Waste]

Hi @skb888, thanks for your interest in LLM Trainer V2!

Is this feature on the roadmap? If so, when it will be ready?

It's on the roadmap. However, it's worth noticing that even in TorchTune they only have official multi-node support for Llama-3.3-70B. So, this feature can only be enabled when we support this model in the future.

It seems that currently Python SDK kubeflow V0.1.0 does not support for peft_config and multi-node, which is also highlighted in the doc

Yes, that's true. But we've supported LoRA fine-tuning in Kubeflow Trainer V2.1.0 and Kubeflow SDK v0.2.0. You can try this feature by downloading the latest release.

[Electronic-Waste]

We're sorry for not updating the docs in time. We'll update it soon

[Electronic-Waste]

/cc @andreyvelich

[andreyvelich]

@Electronic-Waste Please update the docs to indicate that we support LoRA in TorchTuneConfig.
For multi-node training we should discuss whether we want to enable it in torchtune given their migration to the torchforge recently: meta-pytorch/torchtune#2883 (comment)
cc @felipemello1

[felipemello1]

hey! Torchtune does support multi-node, however we dont actively maintain it anymore :/. There shouldnt be new features added to tune. Our focus is now on TorchForge, where we have scalability as a first class citizen. Thanks for the interest!

[Electronic-Waste]

@felipemello1 Thanks for the response. Does torchtune support multi-node modes for every model? I only find that Llama-3.3-70B has multi-node configuration: https://github.com/meta-pytorch/torchtune/blob/main/recipes/configs/llama3_3/70B_full_multinode.yaml

[felipemello1]

@Electronic-Waste , the config is just an example, since the number of possibilities is combinatorial. It should work for every model.