RedPajama-Data-v2: an Open Dataset with 30 Trillion Tokens for Training Large Language Models
An open dataset and pipeline code for RedPajama-V2, a 30-trillion-token dataset built from CommonCrawl for training large language models.
RedPajama-V2 includes over 100 billion text documents from 84 CommonCrawl snapshots processed with the CCNet pipeline, of which 30 billion documents carry quality signals and 20 billion are deduplicated. It covers English, German, French, Italian and Spanish; for the annotated and deduplicated part of the dataset the material reports, for example, 14.5 billion English documents totaling 20.5 trillion tokens.
The pipeline has three steps: preparing artifacts, computing quality signals, and deduplication. It runs through a built Docker image and Apptainer, pulls data from an S3 bucket with s5cmd, and needs a fixed PYTHONHASHSEED environment variable for consistent hashing when not run through the provided Docker containers.
It is written in Python, is also published on Hugging Face, and is licensed under Apache-2.0; code for the earlier RedPajama-1T dataset lives on the repository's rp_v1 branch.