LabVLA
A vision-language-action foundation model that turns a vision-language backbone into a real-time robot controller for scientific laboratory tasks, trained with the pi0.5 recipe.
LabVLA fine-tunes a Qwen3-VL-4B-Instruct vision-language model into a robot controller by adding a diffusion-transformer action expert trained with flow matching, following a three-stage recipe: pre-training the backbone on action tokens, post-training the action expert with flow matching under a knowledge-insulation step that stops gradients between the two parts, and finally task-specific fine-tuning. It supports both delta and absolute action modes in the same output vector, for example moving an arm joint by a delta while a gripper's position is given absolutely, and can co-train on additional visual question-answering data alongside its robot demonstrations.
Training efficiency techniques include selective gradient checkpointing of only some modules per stage, fused kernel operations, ZeRO-2 optimizer sharding and exponential moving average parameter offload, which together keep a batch size of 64 running on a single 80GB GPU.
It is written in Python, requires CUDA 12.6 and PyTorch 2.7.1, and is licensed under MIT.