Local Qwen Deployment on vLLM
A demo project that packages a self-hosted LLM chat interface, running Qwen2-1.5B-Instruct or another Hugging Face model behind vLLM with a Streamlit front end, into a single Docker container deployable locally or on an AWS GPU instance.
This project demonstrates deploying a private, self-hosted chat interface for a language model on a developer's own GPU. It serves Qwen2-1.5B-Instruct, or any compatible Hugging Face model, through vLLM as the inference engine, with a Streamlit front end, both running inside one Docker container.
A single entrypoint script starts vLLM first, polls its /health endpoint until the model has loaded, then starts the Streamlit interface; the Streamlit UI listens on port 80 and vLLM's OpenAI-compatible API on port 8000. The repository includes a guide for provisioning an AWS EC2 g4dn.xlarge GPU instance to run it on.
The README also compares this self-hosted approach against calling a hosted API such as OpenAI or Anthropic, on cost, privacy, control and setup effort, as a way of showing when each approach is the better fit.