oMLX
An LLM inference server for Apple Silicon Macs, managed from the menu bar, with continuous batching and a tiered in-memory and SSD key-value cache.
oMLX is an LLM inference server for Apple Silicon Macs, managed directly from the macOS menu bar. It persists key-value cache across a hot in-memory tier and a cold SSD tier, so past context stays cached and reusable across requests even as context changes mid-conversation, which is meant to make local models practical for real coding work with tools such as Claude Code.
It supports continuous batching, pinning everyday models in memory, auto-swapping heavier models on demand, and setting context limits, alongside a menu bar app for managing the server. It installs as a macOS app with in-app auto-update and a bundled CLI shim, through Homebrew, or from source, with optional native custom kernels for faster inference on model families such as GLM-5.2 and MiniMax M3.
It requires macOS 15 or newer and Apple Silicon, and supports an optional MCP integration. It is written in Python.