mlxcel
A Rust CLI and model server for high-performance LLM, VLM, embedding, reranking and audio inference on Apple Silicon and NVIDIA CUDA, executing MLX SafeTensors checkpoints natively with no Python in the request path.
mlxcel runs text generation, multimodal input, embeddings, reranking, speech workloads, continuous batching, prompt caching, speculative decoding and distributed inference in one native runtime, through native MLX C++ bindings rather than a Python request path or a checkpoint-conversion step. Apple Silicon is its primary target, with Linux and CUDA as a secondary target and an experimental, source-build-only AMD ROCm path on Linux.
It serves an OpenAI, Anthropic and Vertex-compatible API (chat completions, completions, responses, embeddings, reranking and audio) alongside native llama-server routes, classified against a frozen manifest of 376 pinned options and routes so nothing is silently ignored. Router mode discovers checkpoints from a model store, directory or INI presets, loads them on demand, bounds the resident set with LRU eviction, and can serve multiple LoRA adapters at once.
The current main branch is v0.7.0 plus unreleased work; installing from source is needed for features not yet in the latest tagged release.