# vector-io

A library that defines one universal file format for vector datasets, so data can be exported from one vector database, re-embedded with a different model, and imported into another, without writing custom conversion code for each pair.

vector-io defines the Universal Vector Dataset Format, a JSON metadata file paired with Parquet files holding vectors and their associated data, as a common interchange point between vector databases. Export and import scripts move data in and out of this format for databases including Pinecone, Qdrant, Milvus, Google Vertex AI Vector Search, KDB.AI, LanceDB, DataStax Astra DB, Chroma and Turbopuffer, with support for several more databases in progress. A separate re-embedding script reads a dataset already in the universal format and produces a new version with vectors from a different embedding model, given the column that holds the text to embed.

It installs with pip, or can be built from a cloned source checkout, and typical use is a command-line export from one database followed by an import into another, both driven from the same on-disk dataset. The project accepts requests for additional database support and documents which integrations are fully supported, partially supported, or still in progress.

Website: https://github.com/AI-Northstar-Tech/vector-io