OpenLLM
A tool for self-hosting open-source large language models as OpenAI-compatible API endpoints with a single command, including a built-in chat UI and deployment paths for Docker, Kubernetes and BentoCloud.
OpenLLM lets a developer run open-source large language models, such as Llama 3.3, Qwen2.5 and Phi-3, or a custom model, as an OpenAI-compatible API with a single command, along with a built-in chat interface and optimized inference backends. It includes a simplified path to enterprise cloud deployment through Docker, Kubernetes or BentoCloud.
It ships with a curated, documented list of supported models, each with its required GPU memory and a ready-made command to start a server for it, so choosing a model and starting it up does not require separately researching its hardware requirements. Once started, the server exposes OpenAI-compatible endpoints on localhost, so it can be used from any framework or tool already built to speak that API.
OpenLLM does not store model weights itself; gated models, such as certain Llama variants, require a separate Hugging Face access token before they can be downloaded and served. It installs with pip.