NeMo Curator
A GPU-accelerated data curation toolkit that builds repeatable pipelines to load, filter, deduplicate and transform large text, image, video and audio datasets for AI training, running the same pipeline on a laptop or a multi-node Ray cluster.
NeMo Curator helps ML engineers and data teams build repeatable, GPU-accelerated pipelines that load, filter, deduplicate and transform large datasets for AI training. The same pipeline can run on a single laptop or scale across a multi-node Ray cluster, including on Slurm-managed HPC clusters, and covers text, image, video and audio workloads.
For text, it supports deduplication, classification, quality filtering and language detection; for images, aesthetic filtering, NSFW detection, embedding generation and deduplication; for video, scene detection, clip extraction, motion filtering and deduplication; and for audio, ASR transcription, quality assessment and word-error-rate filtering. It also includes an inference server that spins up an OpenAI-compatible LLM endpoint inside a pipeline for synthetic data generation, classification and other data-generation workflows, and ships recipes that map to NVIDIA's own training workflows.
It installs with the uv Python package manager and is part of the NVIDIA NeMo software suite for managing the AI agent lifecycle.