VLM3: Vision Language Models Are Native 3D Learners
A NeurIPS 2026 paper and reference implementation showing that standard vision-language models can be trained to do object-level 3D understanding, depth estimation, pixel correspondence and camera pose estimation without special architectures or losses.
VLM3 trains standard vision-language models on 3D tasks by resizing input images to a common focal length, which removes camera ambiguity without adding extra encoders or modules, and by referring to objects or pixels with normalized text coordinates instead of specialized markers or architecture changes. Training uses ordinary supervised fine-tuning on a general-purpose vision-language model.
The paper reports that this approach surpasses SpatialRGPT on object-level 3D understanding, matches other specialized models on metric depth estimation while improving one method's accuracy, surpasses dedicated matching models on pixel correspondence, and matches or surpasses other models on camera pose estimation, arguing that task-specific 3D architectures, losses and data augmentations are not necessary conditions for effective 3D learning. Inference uses the transformers library, with released model weights described as coming soon.