# Local Qwen Deployment on vLLM

A demo project that packages a self-hosted LLM chat interface, running Qwen2-1.5B-Instruct or another Hugging Face model behind vLLM with a Streamlit front end, into a single Docker container deployable locally or on an AWS GPU instance.

This project demonstrates deploying a private, self-hosted chat interface for a language model on a developer's own GPU. It serves Qwen2-1.5B-Instruct, or any compatible Hugging Face model, through vLLM as the inference engine, with a Streamlit front end, both running inside one Docker container.

A single entrypoint script starts vLLM first, polls its `/health` endpoint until the model has loaded, then starts the Streamlit interface; the Streamlit UI listens on port 80 and vLLM's OpenAI-compatible API on port 8000. The repository includes a guide for provisioning an AWS EC2 `g4dn.xlarge` GPU instance to run it on.

The README also compares this self-hosted approach against calling a hosted API such as OpenAI or Anthropic, on cost, privacy, control and setup effort, as a way of showing when each approach is the better fit.

Website: https://github.com/vishakhasadhwani/llm-deployment-demo