continuous-eval
An open-source Python package for data-driven evaluation of LLM-powered applications, with a modular metric library covering retrieval-augmented generation, code generation, agent tool use and classification.
continuous-eval is a Python package for evaluating LLM-powered applications with deterministic, semantic and LLM-based metrics that can be mixed and matched. It measures each module of a pipeline separately with metrics tailored to that module, rather than only scoring a pipeline's final output, and supports probabilistic evaluation of a pipeline.
Its metric library covers retrieval-augmented generation, code generation, agent tool use, classification and other common LLM application patterns. An EvaluationRunner class runs an evaluation over a dataset, and for systems built from multiple modules, each with its own metrics and tests, the library lets a user define those modules and select the corresponding metrics for each one.
It installs from PyPI or from source, needs at least one LLM API key configured in an environment file to run LLM-based metrics, and supports custom metrics, including a class for building an LLM-as-judge metric.