Daft
A high-performance data engine for AI and multimodal workloads that processes images, audio, video and structured data at scale, with built-in operations for running LLM prompts and generating embeddings.
Daft processes images, audio, video, embeddings and structured data together in a single framework, rather than requiring separate tools for multimodal and tabular data. Built-in AI operations run LLM prompts, generate embeddings and classify data using OpenAI, Transformers, or custom models.
It reads from S3, GCS, Iceberg, Delta Lake, Hugging Face and Unity Catalog, and scales from a local machine to distributed clusters on Ray or Kubernetes, with memory management intended to reduce manual configuration.
It installs with pip install daft, requires Python 3.10 or higher, is Python-native with a Rust core, and is licensed under Apache-2.0.