VLM-R1
A research project applying R1-style reinforcement learning to a vision-language model, reporting better out-of-domain generalization than supervised fine-tuning.
VLM-R1 trains Qwen2.5-VL with both a GRPO reinforcement-learning approach and ordinary supervised fine-tuning for Referring Expression Comprehension, a task where a model must locate the region of an image a text description refers to. The project's experiments find that on in-domain test data the two approaches perform similarly over a small number of training steps, but on out-of-domain test data the reinforcement-learned model's reasoning generalizes while the supervised model's performance gradually degrades as training continues.
It supports full fine-tuning, frozen vision modules, LoRA fine-tuning, multi-node training, and multi-image input training for GRPO, and documents how to add a new base model beyond the QwenVL and InternVL families it already supports. A related model trained with this project's approach is reported to lead its category on the Open-Compass Math Leaderboard under four billion parameters.
It is written in Python and distributed under the Apache-2.0 license.