bm25
A Rust crate implementing the BM25 relevance-scoring algorithm as an embedder, scorer and in-memory search engine, for use directly or alongside vector databases such as Qdrant, Pinecone or Milvus.
bm25 is a Rust crate covering BM25 information retrieval at three levels: an embedder that turns text into sparse vectors, a scorer that measures how well a query embedding matches document embeddings, and a lightweight in-memory search engine built on top of both. The sparse embeddings it produces can be used directly or uploaded to a vector database for querying there.
It includes a multilingual tokenizer with stemming, stop-word removal, unicode normalization and language detection, exposes BM25's b, k1 and average-document-length parameters for tuning, and can fit the average document length automatically or work without it. The crate supports parallel batch fitting, compile-time configuration through feature flags, and a companion demo compiles the search engine to WebAssembly for the browser.