Text Embeddings Inference
A toolkit for deploying and serving open-source text-embedding and sequence-classification models, supporting popular families such as FlagEmbedding, GTE and E5.
Text Embeddings Inference runs with no model-graph compilation step, supports Metal for local execution on Macs, ships small Docker images with fast boot times, and does token-based dynamic batching.
It uses optimized transformer code for inference through Flash Attention, Candle and cuBLASLt, loads weights from Safetensors or ONNX, and adds production features such as distributed tracing with OpenTelemetry, Prometheus metrics and gRPC. It supports Nomic, BERT, CamemBERT, XLM-RoBERTa, JinaBERT, Mistral, Alibaba GTE, Qwen2/Qwen3, MPNet, ModernBERT and Gemma3 embedding models, along with reranker and sequence-classification models.
It is written in Rust, runs through Docker, a local install with Homebrew on Apple Silicon, or ARM64 and ROCm builds, and is licensed under Apache-2.0.