pdf-struct-chunker
A pure-Rust command-line tool that splits PDFs into chunks for retrieval-augmented generation by analyzing layout, font size and boldness rather than a fixed token or character count.
pdf-struct-chunker addresses a common problem in RAG pipelines: chunking a PDF by a fixed token or character count cuts through headings and paragraphs, so a vector search later returns fragments with no sense of where they came from. It instead analyzes a PDF's actual layout, including X and Y coordinates, font sizes and bold detection, to find where a heading starts, where a paragraph ends and where a new section begins, and attaches section, heading and page metadata to each resulting chunk.
It runs entirely offline, needs no LLM or API calls, and is written in pure Rust for speed on CPU-only, edge-AI scenarios. A command-line flag set lets a user point it at an input PDF, supply a JSON profile of custom regex rules, choose JSON or JSONL output, and print chunk statistics instead of the chunks themselves.