abstracts-search
A project that indexes more than 200 million academic publication abstracts from OpenAlex into a semantic search engine, using embeddings and a fast vector index.
abstracts-search indexes the publicly available abstracts in the OpenAlex dataset into a single semantic search engine. It generates embeddings for each abstract with a sentence-embedding model, then trains a fast vector index with faiss; the process can be repeated to keep the index synced with OpenAlex's quarterly snapshots.
The project is split across four repositories: this one holds an orchestration Makefile and a helper program; a companion dataset holds the raw embeddings; another repository serves the search interface as a Hugging Face Space; and a fourth holds the trained index, all released under CC0. The search interface maps a query embedding to OpenAlex IDs by semantic similarity and then queries OpenAlex for the associated titles, abstracts and authors, so an internet connection is required at query time. The project also loads a spun-off set of command-line build tools for constructing sidecar indexes.
Building the embeddings from scratch was originally done on a cloud machine with four GPUs over a week; running the search interface alone against an already-synced index needs much less hardware.