Datasets
A lightweight Python library for loading ready-to-use public datasets in one line and for efficiently pre-processing them alongside your own local data.
Datasets is a lightweight library with two main features: one-line data loaders for a large number of public datasets, such as image, audio, text, 3D medical image, video and AI agent trace datasets hosted on the Hugging Face Datasets Hub, and efficient, reproducible pre-processing for those public datasets as well as local data in formats such as CSV, JSON, JSONL, Parquet, HDF5, XML, PNG, JPEG, WAV, MP3 and PDF.
It natively converts data to and from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX and Spark, streams datasets on the fly without downloading them first, reads and writes Hugging Face Storage Buckets for mutable large-scale data, and can load and process AI agent traces such as prompts, tool calls and responses. It uses a zero-copy, memory-mapped Apache Arrow backend so datasets are not limited by available RAM, caches processed results automatically, and supports multi-process parallel processing.
It is maintained by Hugging Face, written in Python, and licensed under the Apache-2.0 license.