NaijaEval
An evaluation library for African-language and code-switching NLP systems, with metrics for dialectal robustness, terminology accuracy and hallucination that mainstream benchmarks do not measure.
NaijaEval provides composable metrics for testing NLP and speech systems on Nigerian English, Yoruba, Igbo, Hausa, Nigerian Pidgin and Swahili, and on code-switching between them. Standard benchmarks such as GLUE or XTREME are built for high-resource languages and do not surface problems specific to these settings.
The library measures things mainstream metrics miss: a code-switch rate, a dialectal robustness score comparing performance on standard versus dialectal input, a terminology preservation rate that weights domain terms such as medical or legal vocabulary more heavily than common words, and hallucination and consistency scores for low-resource translation. It also includes standard measures such as BLEU, chrF, and word and character error rate.
It ships with benchmarks for machine translation and ASR robustness, and built-in domain term lists for medical, legal, financial and customer-support text. It is a Python package installed with pip.