ParqDB
An embedded vector database for larger-than-memory search and analytics on billion-scale multimodal data, using Parquet storage with Arrow-native execution.
ParqDB is an embedded vector database built entirely on Parquet and Arrow, designed for larger-than-memory search and analytics over billion-scale multimodal data.
It searches one billion vectors at high recall using a few CPU cores and a few gigabytes of memory, and stores both source data and vector indexes as standard Parquet files rather than a proprietary format, so indexes can be versioned and shared across engines. Immutable IVF-LVQ indexes can be published to object storage and queried directly from a browser over HTTP Range and WebAssembly, without running a query server. Vector search is expressed as relational SQL operations, so filters, joins and aggregations can combine with vector search in one execution plan, and the same index scales from a single machine up to Spark or StarRocks clusters.
It is written in Python, MIT licensed, and installs with pip.