DocETL
A system for processing large collections of structured and unstructured data with LLMs, letting users describe operations in natural language while it orchestrates, optimizes and parallelizes the pipeline.
DocETL helps process large collections of data with LLMs by letting a user write each operation in natural language, such as pulling out every complaint in a support ticket. It provides operators including map, reduce and filter, orchestrates and parallelizes them across the data, and returns tables that can be queried in a database.
It optimizes pipelines automatically by swapping models, rewriting prompts, decomposing operations and replacing subtasks with code where possible, aiming to improve accuracy and reduce cost. It can be configured through a Python API or a low-code YAML file, and includes DocWrangler, a visual playground for interactive prompt development that can run locally or through its hosted version.
It installs as a Python package and is licensed under MIT.