Adaptive Chunking
A framework that automatically selects the best text-chunking method for each document in a retrieval-augmented generation pipeline, using intrinsic quality metrics instead of one fixed strategy for every document.
Adaptive Chunking selects the best splitting strategy for each document in a RAG pipeline by evaluating multiple chunking methods against a set of intrinsic quality metrics, rather than using one fixed method for every document. Both the chunking methods and the evaluation metrics are modular, so a user can plug in their own splitter or their own metric.
It ships a recursive text splitter with configurable chunk size, overlap, and merging behavior, and metrics such as size compliance, intra-chunk cohesion, and block integrity that can be computed separately from the splitting step. The paper's evaluation covers 33 documents across three domains and roughly a million tokens, with pre-parsed documents and pre-computed coreference clusters included for reproducing the reported tables and figures.
It is written in Python under the MIT license, and its methods are described in a paper accepted at LREC 2026.