Language Model Evaluation Harness
A framework for few-shot evaluation of language models across a large, config-driven library of benchmark tasks, with pluggable model backends, filters and metrics.
The Language Model Evaluation Harness is a framework for few-shot evaluation of language models. Tasks are defined through configuration, with support for Jinja2-based prompt templates, imported prompts from Promptsource, output post-processing, answer extraction and multiple generations per document, and it includes the task groups behind the Open LLM Leaderboard.
Model backends, filters, metrics and aggregations can be registered from a separate package instead of requiring a fork of the project, either through a declared entry point for automatic discovery or by pointing the tool at a local plugin module. Its command-line interface is organized into subcommands for running evaluations, listing available tasks, and validating a task configuration, with YAML configuration file support.
The base package no longer bundles heavy dependencies such as transformers or torch; model backends including Hugging Face Transformers, vLLM and SGLang are installed separately as extras, and there is prototype support for multimodal text-and-image tasks.