Thinking with Visual Primitives — PyTorch Implementation
An unofficial PyTorch reproduction of DeepSeek's method for teaching multimodal LLMs to reason with bounding boxes and points interleaved directly in their chain of thought.
This project implements a multi-stage training pipeline that teaches multimodal LLMs to reason with bounding boxes and points as first-class "thought units," interleaving spatial coordinates within chain-of-thought text to close what the authors call the Reference Gap in visual reasoning.
Training runs in three stages: pretraining a base model to output the visual-primitive format, specialized supervised fine-tuning of separate box-grounding and point-grounding experts, and on-policy distillation of both experts into one unified model. Because of compute constraints, every stage in this reproduction uses LoRA fine-tuning rather than full-parameter training, so the results are presented as a feasibility check rather than a state-of-the-art claim.
Pretrained checkpoints for each stage, built on Qwen2VL-2B, are published on Hugging Face. It is written in Python with PyTorch and is MIT licensed.