distillery
A Python library and CLI that turns source documents such as PDFs into instruction fine-tuning datasets, grounding each generated example in a specific chunk of the source, grading it with an LLM judge, and deduplicating by embedding rather than exact text.
Distillery builds a fine-tuning dataset from real source documents (PDFs, text files or URLs) rather than requiring one to be curated by hand. Every generated example is grounded in a specific chunk of the source or explicitly flagged as free-form, graded by an LLM judge against its reference context, and deduplicated by the embedding of its instruction text rather than a string match.
The pipeline chunks source text with sentence-aware overlap, proposes seed questions per chunk plus some free-form ones from a description, generates answers from the originating chunk, filters by embedding-based diversity and by content-word overlap between answer and source as a hallucination check, and can optionally produce DPO preference pairs by keeping only cases where a deliberately weaker answer loses to a stronger one.
Every example carries provenance back to its source chunk, and a manifest.json records what went into the dataset, what was rejected, and why. Output formats include JSONL, OpenAI-style messages, Hugging Face datasets, flat instruction/output pairs and DPO pairs, and it can run as a CLI, a library, or through an optional MCP server.