LLM Production Platform
A CPU-first reference platform showing the systems around LLM inference: an OpenAI-compatible gateway, a deterministic eval CLI, an experiment registry and a RAG toolkit.
This platform demonstrates the systems that surround LLM inference, such as streaming, API compatibility, observability, evaluation, reproducibility and retrieval, without reimplementing the inference engine itself. It has four independently usable surfaces: an OpenAI-shaped FastAPI gateway in front of a separately running llama-server, a deterministic evaluation and regression CLI, an immutable local experiment registry, and a deterministic local retrieval-engineering toolkit; the serving runtime does not import the other three.
Each surface defines explicit persistence, failure, privacy and reproducibility contracts: streaming preserves backpressure and cancellation, metrics use bounded-cardinality labels, experiment artifacts are immutable and checksummed, and RAG inputs, chunks, indexes and citations are fingerprinted. Default tests need no model, GPU, Docker daemon, hosted API, database or secret.
The project describes itself as a reference and portfolio project, not a replacement for vLLM, SGLang, managed model platforms, distributed schedulers or production vector databases.