Xorbits Inference
A library for serving language, speech-recognition and multimodal models with a single command, exposing them through a unified, OpenAI-compatible inference API on the cloud, on-premises or on a laptop.
Xorbits Inference (Xinference) is a library for serving language, speech-recognition and multimodal models. It deploys and serves either a project's own models or built-in models with a single command, on the cloud, on-premises, or on a laptop, through one unified, OpenAI-compatible inference API.
It integrates with Xagent, an agent platform, for dynamic planning, tool use and autonomous multi-step reasoning on top of served models, and with platforms such as Dify and FastGPT for building LLM applications. Recent releases add automatic batching of concurrent requests for higher throughput, a new llama.cpp Python binding with continuous batching, distributed inference across multiple workers, and shared KV cache across vLLM replicas, alongside a growing list of built-in models for OCR, image generation and editing, text-to-speech and other tasks.
It is written in Python and released under the Apache-2.0 license.