GRACE-VLM
A quantization-aware distillation method that shrinks an 8B vision-language model into a 2B student and deploys it as genuine packed INT4 weights, retaining most of the teacher's accuracy.
GRACE-VLM distills an 8B-parameter vision-language model into a 2B-parameter student using confidence-aware knowledge distillation combined with group-wise quantization-aware training. The released 2B BF16 student scores 76.7 averaged across seven vision-language-model benchmarks, slightly above the 76.3 scored by its 8B teacher, and a quantized version of that student scores 75.0, retaining 98% of the BF16 average. The primary deployment checkpoint is a genuine AWQ-packed INT4 build with real packed weight tensors, distinct from a similarly named BF16-on-the-INT4-grid research checkpoint that does not provide INT4 memory or speed benefits through plain loading.
The project provides a hosted demo Space for trying the BF16 model online on free GPU, a standard Transformers checkpoint for further research or fine-tuning, and a released Docker image, with a Dockerfile to build it locally, for running the packed INT4 model, which needs an extra kernels package installed for fused CUDA kernels or otherwise falls back to a slower PyTorch dequantization path. It targets a Linux host with an NVIDIA GPU for real inference, though the image can be built and smoke-tested under emulation on Apple Silicon.