mini-verl
mini-verl adapts the verl reinforcement-learning framework to run PPO, GRPO and on-policy distillation training on a single consumer NVIDIA GPU, reusing existing verl configs and Hydra overrides.
mini-verl brings PPO, GRPO and on-policy distillation training, normally run on a verl cluster, down to a single consumer NVIDIA GPU. After installing a CUDA-matched PyTorch build, a first experiment samples reward-labeled prompt data, then runs a dry-run and a full training run using an example Hydra configuration, all against a small Qwen3-0.6B model with a simple length-based reward.
An existing verl experiment's own configuration tree and Hydra overrides can be passed in directly, so a user does not need to rewrite reward or training logic already built for verl. The project also supports resuming interrupted training runs and exporting trained adapters, and separately documents native recipes for SFT, DPO and offline knowledge distillation. Guides help with choosing a model and memory settings for a given GPU, and with migrating a larger existing verl setup.