GPT Server
An open-source framework, no longer maintained, for production deployment of large language, embedding, reranking, speech, and image and video generation models behind a single OpenAI-compatible API port.
GPT Server built on FastChat to expose chat, embedding, reranking, text moderation, speech-to-text, text-to-speech and Stable Diffusion image and video models through OpenAI-compatible API endpoints, so any project written against the OpenAI API could point at it instead. It supported multiple inference backends, including Hugging Face Transformers, vLLM, LMDeploy and SGLang, and could route several models through the same port with automatic scheduling.
It also implemented the Cohere-compatible rerank endpoint used by tools like Dify. The project's README states it stopped receiving updates and bug fixes in May 2026 because of conflicting dependencies between vLLM, SGLang and LMDeploy versions, and recommends migrating to the official vLLM or SGLang images directly, or to the vllm-omni project for ASR, TTS and Stable Diffusion workloads.