mlx-serve
A local inference server for Apple Silicon that hot-swaps MLX models on demand, loading one text, vision, embedding, TTS, STT, or decision model at a time behind an OpenAI-compatible API.
mlx-serve is a local inference server for Apple Silicon that loads exactly one MLX model at a time and hot-swaps it automatically when a request names a different configured model, keeping memory use within unified-memory limits. It covers six model types: text, vision, embeddings, text-to-speech, speech-to-text, and a typed decision model called Laya.
Text and vision models run as isolated subprocesses so the operating system reclaims all their memory cleanly on unload, unlike running everything in one process, which the project says causes memory fragmentation over long sessions. It exposes an OpenAI-compatible API that works as a drop-in with LiteLLM or any OpenAI SDK, and is designed to be headless and scriptable rather than requiring a GUI.
Setup generates a models.yaml configuration file naming the models to serve, then a mlx-serve start command launches the server on port 8095.