LLM Evaluation Framework
An open-source framework that benchmarks GPT-4, Claude, Gemini, Mistral, and Llama side by side on the same prompts, measuring accuracy, latency, per-token cost, a hallucination signal, and reasoning quality.
LLM Evaluation Framework runs several models against the same prompts and reports five metrics: accuracy (exact, normalized, multiple-choice, and fuzzy match), latency percentiles from p50 to p99, per-token cost across more than 15 priced models, a hallucination rate from linguistic signal analysis, and chain-of-thought reasoning quality.
It ships built-in support for the MMLU and TruthfulQA benchmarks, accepts custom CSV, JSON, or Hugging Face datasets, and exposes the results through a CLI with several subcommands, a FastAPI REST API with OpenAPI docs, a five-page Streamlit dashboard, an async Python API, and PDF report generation.
Evaluation runs in parallel with an async semaphore, results persist to SQLite, and the project ships a Docker Compose setup and a test suite run under GitHub Actions across multiple Python versions.