Retrieval augmented generation with quantized LLM
A set of retrieval-augmented-generation demos that run entirely offline on CPU with quantized open models such as DeepSeek, Llama, Qwen, Mistral and Gemma.
This repository holds retrieval-augmented-generation demos built around quantized large language models, including DeepSeek, Qwen, Aya-Expanse, Mistral, Gemma, Llama and Phi. The demos run on CPU with acceptable inference time and work fully offline, so they can be deployed in an air-gapped environment.
They support applying a propositionizer to document chunks, reranking retrieved passages, and hypothetical document embedding. Embedding, reranking and generation models are downloaded and configured through a YAML file, and the demos use Phoenix for tracing LLM calls during experimentation. It is licensed under Apache-2.0.