Benchmarking Large Language Models for Bio-Image Analysis Code Generation
A fork of the HumanEval evaluation harness adapted to benchmark how well large language models generate correct Python code for bio-image analysis tasks.
This project adapts the HumanEval evaluation harness, originally built for the paper Evaluating Large Language Models Trained on Code, to the bio-image analysis domain. It defines test cases for evaluating the functional correctness of generated bio-image analysis Python code and reports the pass rate, the probability that generated code passes its given unit tests, across 15 compared LLMs, both overall and per task.
It supports running the benchmark against OpenAI models, Google Gemini models, models available through the Helmholtz blablador service for German academics, and local models served through Ollama. The program deliberately executes untrusted, model-generated code, so the authors strongly encourage running it only inside a proper security sandbox.