rag-omni
An example retrieval-augmented-generation project demonstrating question answering over an internal document set using a locally deployed large language model.
rag-omni is a demonstration project for retrieval-augmented generation, built around the scenario of answering questions over a set of internal government policy documents using a locally deployed, open large language model rather than a hosted API. It covers preprocessing raw Word and PDF documents into structured, retrievable chunks, using scripts for straightforward text documents and pointing to external OCR tooling for image-only PDFs, and building a retrieval index from the result.
For serving the model, it uses the LLaMA-Factory project's inference server accelerated with vLLM, exposing an OpenAI-compatible API, and includes test scripts for functional and load testing. An example nginx configuration shows how to load-balance requests across several model instances running on different GPUs in the same machine. The author notes explicitly that this is a demonstration of the overall RAG process rather than a production-grade system, since the open models and retrieval methods used are not tuned for production-level accuracy.