FineWeb
Large-scale Pre-trained Dataset
FineWeb is a large-scale high-quality English web text dataset released by Hugging Face, containing 15 trillion tokens, selected from 96 Common Crawl snapshots, specifically designed for pre-training large language models. It employs a multi-stage data quality filtering pipeline (URL filtering, text extraction, deduplication, model quality scoring), surpassing similar datasets like C4, Dolma, and RedPajama.
Dataset Highlights
Industrial-grade high-quality corpus specifically designed for pre-training large language models
Ultra Large Scale
Contains 150 trillion tokens, making it the largest open English web text dataset currently available, providing ample corpus for training GPT/LLaMA level large language models.
Quality-First Pipeline
Employs a multi-stage data cleaning process: URL filtering, trafilatura text extraction, MinHash deduplication, model quality scoring, ensuring high quality for each data point.
Surpassing Similar Datasets
In LLM pre-training benchmark tests, FineWeb's performance significantly surpasses mainstream open-source datasets such as C4, Dolma, and RedPajama.
CommonCrawl Source
Extracted from 96 complete Common Crawl snapshots from 2013 to 2024, covering a wide range of web content and time spans.
Open License (ODC-By 1.0)
Utilizes the Open Data Commons Attribution license, allowing commercial use, modification, and redistribution with proper attribution.
Hugging Face Ecosystem
Seamlessly integrates with the Hugging Face datasets library, supporting streaming data loading, allowing training to start without downloading all data.
Applicable Scenarios
From model training to academic research, fully supports the LLM development process
LLM Pre-training
Train large language models at GPT and LLaMA levels from scratch, providing massive high-quality English corpus support
Continuous Pre-training
Continuously pre-train existing foundational models for domain adaptation, enhancing language understanding capabilities in specific fields
Data Quality Research
Research the design and effects of data filtering pipelines, exploring the impact of data quality scores on model performance
Benchmarking and Evaluation
Compare the impact of different dataset qualities on model performance, providing experimental basis for dataset selection
Data Preview
The following is a JSONL format example of the FineWeb dataset, with each line containing text content, source URL, token count, and quality score
{"text": "Artificial intelligence continues to reshape industries across the globe...", "url": "https://example.com/ai-article", "token_count": 2847, "quality_score": 0.94}
{"text": "The latest advances in renewable energy technology promise...", "url": "https://example.com/energy-tech", "token_count": 1523, "quality_score": 0.91}
{"text": "Understanding neural network architectures requires...", "url": "https://example.com/nn-guide", "token_count": 4102, "quality_score": 0.97}
3 Steps to Get Started Quickly
Quickly integrate FineWeb into your LLM development process from browsing to training
Browse Datasets
Browse FineWeb datasets on the Ace Data Cloud platform and view details such as metadata, licensing agreements, and data scale.
Stream Load or Download
Stream load data partitions using the Hugging Face datasets library, or download specific CC snapshot subsets on demand.
Integrate into Training Pipeline
Feed the data into your tokenizer and training pipeline, supporting mainstream deep learning frameworks such as PyTorch, JAX, and TensorFlow.
Start Exploring FineWeb Data
150 trillion tokens of high-quality English corpus, open license, available immediately. Whether you are training the next generation of large language models or researching data quality pipelines, FineWeb is your ideal choice.