Long-Form Factuality in Large Language Models
This is the official code release for a research paper on benchmarking long-form factuality in large language models, including a fact-seeking prompt set and an automatic evaluator for long-form model responses.
This repository is the official code for the paper Long-form factuality in large language models. It provides LongFact, a set of 2,280 fact-seeking prompts that each require a long-form response, and the Search-Augmented Factuality Evaluator (SAFE), which automatically evaluates a model's long-form responses for factual accuracy. It also implements F1@K, an extension of the F1 score to long-form settings that uses recall based on a human-preferred response length.
An included experimentation pipeline benchmarks OpenAI and Anthropic models by generating responses to the LongFact prompts and then scoring them with SAFE, and separately evaluates SAFE itself against human annotations from FActScore.
It is written in Python, installs into a conda environment from a requirements file, and requires API keys for the model providers being benchmarked.