Model2Vec
A technique and library for distilling any sentence-transformer model into a small, fast static embedding model, cutting size by up to 50 times and inference time by up to 500 times for a small drop in accuracy.
Model2Vec turns an existing sentence transformer into a small, fast static embedding model. It reduces model size by up to a factor of 50 and speeds up inference by up to 500 times on CPU, at the cost of a small drop in performance, and its best model is reported to be the most performant static embedding model of its kind.
Pre-trained flagship models can be loaded directly from the Hugging Face hub for tasks such as text classification, retrieval and clustering, or a user can distill their own Model2Vec model from any Sentence Transformer model in roughly 30 seconds on a CPU. After distillation, a classification model can be fine-tuned on top of the result, or on top of a pre-trained model directly.
The base package's only major dependency is numpy, with optional extras for distillation and training. It is written in Python and released under the MIT license.