kibble
A fast Rust command-line tool that pulls in data from local files, git repositories, websites, Hugging Face datasets, and file hosts, cleans and classifies it, and builds deduplicated, leakage-safe chat-format JSONL datasets for training and retrieval.
kibble acquires knowledge and data at scale from almost any source -- local files, git repositories, websites, Hugging Face datasets, file hosts, and archives -- extracts and cleans the content, classifies and catalogs it, and builds chat-format JSONL datasets meant as raw material for training and retrieval. It can also run as an HTTP service so bots or automation can submit links for ingestion, and it ingests agent skills the same way it ingests data.
Heavy lifting such as model training, OCR, and embeddings runs in external backends that kibble routes to; the core pipeline (clean, index, search, build, eval, pack, crawl) works fully offline, falling back to lexical BM25 search when no embedding backend is configured, while ask, bench, cluster, and extract need an LLM or embedding endpoint. A site crawler is robots-respecting, polite, and SSRF-guarded.
Prebuilt binaries are available for Linux, macOS, and Windows, and a kibble init command scaffolds a starter project with a config file and sample docs.