DataTerminology.comNEW
Also known as: Colossal Clean Crawled Corpus
A cleaned, deduplicated corpus derived from Common Crawl used widely for text-to-text pretraining (e.g., T5).
DESCRIPTION
C4 is a large-scale text corpus created by cleaning and filtering Common Crawl, popularized by the T5 paper. It removes low-quality and non-English content and performs deduplication. Distributed via TensorFlow Datasets, C4 standardizes preprocessing for reproducible training. It became a common baseline for text-to-text and language modeling tasks. Variants adjust filtering rules to trade off size and quality.
ALTERNATIVE DEFINITIONS
TensorFlow Datasets
C4 (Colossal Clean Crawled Corpus) is a cleaned version of Common Crawl designed for large-scale language model pretraining.
Highlights the cleaned Common Crawl origin and pretraining purpose.
arXiv
We introduce the Colossal Clean Crawled Corpus (C4), a cleaned version of Common Crawl for text-to-text pretraining at scale.
Anchors C4’s role in the T5 text-to-text framework.
RELATED TERMS
Common Crawl T5 Text-to-Text Transfer Language Modeling Entity Matching Scrubbing TFDS Tokenizer WebText Pretraining
COMPETING TERMS
OpenWebText The Pile
RELATIONS
- Broader: Dataset
- Derived from: Common Crawl
KNOWLEDGE AREAS
AI Data Science Data Management
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- TensorFlow - Official C4 dataset card
- arXiv - T5 paper introducing C4
TOP QUESTIONS
Why was C4 created?
C4 standardizes a cleaned subset of Common Crawl for reproducible large-scale pretraining, reducing noise and non-English content to improve model quality.
Category: Origin
How is C4 different from Common Crawl?
C4 is a heavily filtered, deduplicated subset designed for ML training. Common Crawl is the raw, broad web archive with minimal content restrictions.
Category: Comparison
What is C4’s role in T5?
C4 underpins T5’s text-to-text pretraining, enabling a unified approach across diverse NLP tasks with standardized data cleaning.
Category: Context
What are typical preprocessing steps beyond C4?
Additional deduplication, PII redaction, and domain reweighting are often applied depending on compliance and performance needs.
Category: HowTo
What are risks of training solely on C4?
Domain gaps, residual noise, and license ambiguities can affect downstream behavior. Complement with curated datasets and compliance checks.
Category: Risks
Is C4 multilingual?
No. C4 focuses on English content. For multilingual needs, use alternative corpora or multilingual variants.
Category: Scope
How big is C4?
C4 contains hundreds of gigabytes of English text after filtering, sufficient for substantial language model pretraining.
Category: Scale
What datasets compete with C4?
OpenWebText and The Pile provide alternative pretraining corpora with different curation philosophies and domain mixes.
Category: Alternatives
Where can I get C4?
It’s available in TensorFlow Datasets and mirrored by other dataset hubs, with variants and shards for scalable ingestion.
Category: HowTo