TinyToT
A local, privacy-first inference server that separates fact storage from language composition, retrieving facts from plain markdown files with a small keyword index instead of relying on a large model's memorized weights.
TinyToT answers questions, reasons, generates code, handles documents and holds multi-turn conversations without model weights, a GPU or a network connection. It is built on the idea that a large language model does two different things, storing facts in its weights and composing retrieved facts into an answer, so TinyToT keeps facts as plain markdown files retrieved through a five-head TF-IDF index with BM25 re-ranking, leaving only open-ended generation to a small parameter-based model.
It exposes an Ollama-compatible chat-completions endpoint, and new facts can be added by dropping a markdown file into a knowledge folder and reloading, without retraining anything. The project reports internal benchmark results across routing accuracy, factual question answering, held-out reasoning, code generation across several languages, document extraction, and a small puzzle-solving game, among other tasks.
It is written in Python and released under the MIT license.