RedPajama
Data-1T
RedPajama-Data-1T is an open reproduction version of the LLaMA training dataset created by Together AI, containing 1.2 trillion tokens from seven data sources: CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, and StackExchange, licensed under Apache 2.0, supporting transparent and reproducible large language model training.
Dataset Highlights
Open and transparent trillion-scale pre-training dataset to support research and development of large language models
Trillion Token Scale
Contains 1.2 trillion tokens, fully matching the original training data scale of LLaMA, providing ample data support for pre-training large models.
Seven Data Sources
Covers CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, and StackExchange, encompassing diverse fields such as web pages, code, encyclopedias, and academia.
Transparent Processing Workflow
Complete documentation of data processing and filtering pipeline is publicly available, with every operation traceable and auditable, ensuring full transparency of data sources and quality.
Apache 2.0 License
Uses a permissive Apache 2.0 open-source license, supporting academic research and commercial applications without worrying about licensing restrictions.
Quality Filtering
Each data source is processed using domain-specific cleaning rules, including deduplication, language detection, quality scoring, and other multi-dimensional filtering.
Fully Reproducible
Complete methodology and processing workflow have been open-sourced, allowing researchers to reproduce, customize, and extend the dataset to meet their needs.
Applicable Scenarios
Covers various large language model development scenarios from model pre-training to data research
LLM Pre-training
Train large language models from scratch using validated data recipes to replicate LLaMA-level training results
Data Ablation Experiments
Study the impact of different data sources on model performance, quantifying the contribution and importance of data from various fields
Curriculum Learning
Design multi-stage training curricula across data domains, optimizing data mixing ratios and training scheduling strategies
Model Comparison
Conduct fair comparisons of model architectures using standardized training data, eliminating interference from data differences
Quick Start
Quickly access the RedPajama dataset via API
import requests
url = "https://api.acedata.cloud/datasets/redpajama"
headers = {
"Authorization": "Bearer YOUR_API_TOKEN",
"Content-Type": "application/json"
}
params = {
"source": "wikipedia",
"limit": 10
}
response = requests.get(url, headers=headers, params=params)
data = response.json()
# Print the returned data entries
for item in data.get("data", []):
print(item.get("text", "")[:200])
print("---")
3 Steps to Get Started Quickly
From registration to usage, you can start accessing trillion-scale pre-trained data in just a few minutes
Register an Account
Register your Ace Data Cloud account at platform.acedata.cloud to quickly complete developer onboarding.
Get API Key
Create your API key in the console for authentication and data access authorization.
Start Using the Dataset API
Access the RedPajama-Data-1T dataset via the API, querying and downloading pre-trained data from seven major data sources as needed.
Start Exploring the RedPajama Dataset
Open license, trillion-scale, completely transparent. Whether you are training large language models or conducting data research, RedPajama-Data-1T is the ideal choice.