AI-Lake Format
A vector-native data lakehouse file format, written in Rust and built on the Apache Iceberg specification, that stores tabular data, embeddings and an HNSW vector index together in one self-contained Parquet-extended file.
AI-Lake places tabular data, embeddings and an HNSW index inside a single file at the object-storage layer, with the vector index stored in the file's footer past the final Parquet magic bytes, so standard Parquet readers such as Spark, Trino, DuckDB, Athena and Snowflake can read AI-Lake tables without any plugin, while the vector search capability is additive rather than a fork of the format. Apache Iceberg's snapshot isolation applies to vector search the same way it applies to SQL queries, so time travel, rollback and concurrent writers work without eventual-consistency gaps or index-rebuild windows.
Each file records its vector centroid and radius in the Iceberg manifest, letting a query eliminate geometrically distant files before opening a single Parquet file, which the project reports as skipping 95 to 99 percent of objects on tables with thousands of files. NVIDIA cuBLAS and AMD hipBLAS are loaded at runtime through dynamic linking rather than compiled in, so one release binary auto-selects GPU acceleration on CUDA or ROCm machines and falls back to SIMD on CPU-only machines with no recompilation.
The write and search path is pure Rust, with zero-copy PyArrow results for Python and a C-ABI bridge for Spark, Trino and Flink.