Semantic Search Evaluation Framework
A framework for evaluating semantic search across a developer's own datasets, comparing embedding models, BM25 and other backends with metrics such as Accuracy@k and NDCG@k.
Semantic Search Evaluation Framework evaluates semantic search across custom datasets, metrics and embedding backends. A user supplies a document set and a set of test queries, each pointing at the single correct document, and configures the models and metrics to compare through a YAML file.
It supports HuggingFace and OpenAI embedding models alongside BM25, computes metrics including Accuracy@k, NDCG@k and latency, and can generate test queries automatically through OpenAI or a naive keyword-extraction method when a dataset has none. Results can be visualized with seaborn and matplotlib. It is written in Python and MIT licensed.