RAG (Retrieval-augmented generation) ChatBot
A conversation-aware chatbot that answers questions using retrieval-augmented generation over a folder of Markdown files, running entirely on a local language model.
This project combines llama.cpp and Chroma to build a RAG chatbot that answers questions from a collection of Markdown documentation files. A Memory Builder component loads the Markdown pages, splits them into smaller sections using a locally reimplemented version of a recursive character text splitter, embeds them with Sentence Transformers, and stores the embeddings in a Chroma database.
When a user asks a question, the chatbot first has a language model rewrite the question for better retrieval, then retrieves the most relevant sections and uses them as context to generate an answer with a local language model. It remembers previous turns of the conversation and factors that history into later answers. It has been tested on Ubuntu with an NVIDIA GPU and on Apple Silicon Macs.