# Token Reducer

A local-first context-compression pipeline for Claude Code that uses hybrid BM25 and embedding retrieval plus AST-aware code chunking to reduce how many tokens of a codebase are sent to the model.

Token Reducer indexes a codebase and, at query time, retrieves only the context relevant to the current request instead of sending an entire file or codebase to the model. It combines BM25 keyword search with vector embeddings in a hybrid retrieval step, chunks code using tree-sitter based AST parsing rather than plain text splitting, and reranks results before handing them back, aiming to cut token usage substantially while keeping the reasoning entirely on the local machine with no API calls.

It installs as a Claude Code plugin through the plugin marketplace command or by cloning it into the Claude plugins folder, and an optional set of dependencies adds neural sentence-transformer embeddings, fast approximate nearest-neighbor search through hnswlib or faiss, and tree-sitter grammars for languages including Python, JavaScript, TypeScript, Go, Rust, Java, C, C++ and Ruby; without them it falls back to hash-based embeddings.

It is written in Python and released under the MIT license.

Website: https://github.com/Madhan230205/token-reducer