TokenPack-RAG
A tool that packs long documents, codebases and PDFs into a token-budgeted, evidence-dense context file using local embeddings and scoring, without calling an LLM during packing, then hands the result to any model or agent.
TokenPack-RAG selects the most useful parts of long documents, code, PDFs, tables and folders to fit inside a fixed token budget, aiming to avoid both the cost of pasting everything into a long-context model and the blind spots of plain top-k retrieval. It splits input into chunks that respect headings, paragraphs, code blocks and semantic shifts, scores each chunk for evidence value using semantic similarity, keyword support, position and structural signals, and then greedily fills the token budget with the highest-value chunks.
The pipeline runs entirely with local embeddings and does not call an LLM during packing; the output is a plain Markdown context file that can be given to any model or agent, including through an MCP integration for agents working with large local workspaces. An optional LLMLingua or LongLLMLingua pass can compress the selected context further.
It installs through pip with optional extras for PDF, Office and MCP support, and uses sentence-transformers/all-MiniLM-L6-v2 as its default embedding model.