LangTest
A testing library that generates and runs more than 60 kinds of tests for NLP and LLM models, covering robustness, bias, fairness, toxicity and factuality, with automatic training-data augmentation.
LangTest is a Python library for testing natural language processing and large language models. With a single line of code it generates and executes more than 60 distinct types of tests covering robustness, bias, representation, fairness and accuracy, and can automatically augment training data based on test results for supported models.
It supports popular NLP frameworks for named entity recognition, translation and text classification, including Spark NLP and Hugging Face Transformers, and supports testing LLMs from OpenAI, Cohere, AI21, the Hugging Face Inference API and Azure OpenAI for tasks such as question answering, toxicity, clinical tests, legal support, factuality, sycophancy and summarization.
Usage centers on a Harness object that is configured with a task and a model, then generates test cases, runs them and produces a report. The project ships benchmark datasets for question answering, summarization and other evaluation scenarios.