🕸️Common Crawl

NEW

A nonprofit organization that crawls the web and freely provides open web crawl datasets for research and model training.

DESCRIPTION

Common Crawl maintains an open repository of web crawl data, published regularly as WARC files. The corpus spans petabytes of multilingual content and is widely used for NLP pretraining and web research. Data is accessible via public cloud buckets and indexes. Derivative corpora like C4 and The Pile build on Common Crawl with additional filtering and cleaning. Ethical use requires attention to licensing, sensitive content, and robots guidelines.

ALTERNATIVE DEFINITIONS

Common Crawl

Common Crawl is a nonprofit organization that crawls the web and freely provides its archives and datasets to the public.

Concise mission statement highlighting open access to crawl data.

Source

AWS Open Data

The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling.

Emphasizes the corpus scale and longevity via AWS Open Data registry.

Source

RELATED TERMS

Web Crawling WARC C4 Dataset The Pile Open Dataset CC-MAIN Robots.txt Language Modeling Scrubbing Public Dataset

COMPETING TERMS

Internet Archive OpenWebText

RELATIONS

• Derivations: C4 Dataset

KNOWLEDGE AREAS

Data Services Data Engineering AI Data Management

TARGET AUDIENCE

Data Engineer AI Engineer Data Scientist Data Architect

EXTERNAL SOURCES

• Wikipedia - Common Crawl
• Common Crawl - Dataset formats and access info
• AWS Open Data - Public dataset listing and access

TOP QUESTIONS

How often does Common Crawl publish new data?

Common Crawl typically releases new crawls on a regular cadence (often monthly). Each release is labeled CC-MAIN with a timestamp and includes WARC, WAT, and WET files.

Category: Process

How do I access Common Crawl data?

Data is available via public cloud buckets and through the AWS Open Data Registry. You can stream or download WARC files and use indexes to locate relevant pages.

Category: HowTo

What are WARC files in Common Crawl?

WARC is a standard container format for crawled web content and metadata. It packages HTTP requests, responses, and payloads for archival and analysis.

Category: Concept

How is Common Crawl used in AI?

It’s a primary source for large-scale text pretraining. Datasets like C4 and The Pile apply cleaning and filtering steps to derive higher-quality corpora.

Category: Usage

What are risks in using Common Crawl?

Content may include sensitive, copyrighted, or low-quality text. Thorough filtering, deduplication, and compliance checks are crucial for responsible use.

Category: Risks

What is the difference between Common Crawl and Internet Archive?

Common Crawl focuses on open datasets for analysis, while Internet Archive emphasizes web preservation and browsing (e.g., Wayback Machine). Use cases and access patterns differ.

Category: Comparison

How do I filter Common Crawl for language or domain?

Use index files and metadata (WAT/WET) to target languages, mime types, or hostnames, then process selected WARC records for your pipeline.

Category: HowTo

Is Common Crawl data licensed for commercial use?

Datasets are openly available, but downstream use must respect source site licenses and applicable laws. Review Common Crawl terms and perform due diligence.

Category: Compliance

What compute patterns suit Common Crawl processing?

MapReduce, Spark, or Beam can scale WARC processing. Use distributed storage, pagination of index queries, and content-type filters to control costs.

Category: Engineering