Fine-tuning for 4 legged robot

[Harshnigam6]

Hi, I am an incoming Phd student at Memorial University of Newfoundland. My focus will be on robotics and I came accross this amazing work.
I am thinking of using the Pie model to operate a 4 legged robot like unitree quadruped robot. The main project I am going to work on is actually a mobile 4 legged robot with an attached robot arm, but since this model has not been trained on any 4 legged robot , so I am very curious first to see if I can make this model work for a quadruped robot.

I imagine that a direct fine-tuning would not be very useful as the current inputs are specific for arm/wheeled robot joints control, so building on the idea of pali-gemma, Token‑unification: I will linearly project the joint space of the new robot joints to the current embedding size and simply prepended to the sequence that the encoder sees. Based on pali-gemma paper, this would add only an extra 3-5 million parameters.

I wonder if the latent physics rules learnt from arm and wheel robot will transer to a 4 legged robot. For the output I am still researching how I can augment the final output layer.

I am looking into following datasets for fine-tuning:

1. AMP‑in‑Real “German‑Shepherd” set
2. EAGLE / CEAR
3. QUARD
4. TAIL (Terrain‑Aware Multi‑Modal SLAM)
5. And then finally recording data from my own robot which is the target platform.

I was wondering if your team has any intuition on this or if you could provide me any insights into my approach. Very exciting repo and I really appericiate you opening it for researchers and enthusiasts!