# ChatWeb

A tool that crawls web pages or reads PDF, DOCX and TXT files, embeds the extracted text, and answers questions or produces summaries grounded in that content.

ChatWeb crawls a given web page or extracts text from a PDF, DOCX or TXT file, then uses an embedding API based on GPT-3.5 to generate a vector for each paragraph and stores those paragraph-vector pairs in a vector database, following an approach similar to existing chat-with-your-document tools. When a user asks a question, ChatWeb generates keywords from that question, embeds the keywords rather than the raw question to improve match accuracy, performs a nearest-neighbor search over the vector database, and passes the most similar passages to GPT-3.5's chat API to produce an answer or a summary grounded in that retrieved content.

Because only the passages judged most relevant are sent to the chat model, the approach works around the model's context-length limit rather than needing to fit an entire document into one prompt.

It runs either from a manual Python installation, with an OpenAI API key set in a configuration file, or through a bundled Docker Compose setup that exposes a browser interface, and supports switching its interface language. It is written in Python and released under the MIT license.

Website: https://github.com/SkywalkerDarren/chatWeb