Common Crawl Foundation

non-profit

Verified

https://commoncrawl.org

AI & ML interests

Crawled data and metadata

Recent Activity

malteos updated a bucket 3 days ago

commoncrawl/commoncrawl

parismic updated a dataset 3 days ago

commoncrawl/statistics

malteos new activity 5 days ago

commoncrawl/commonlid-results:Add supported_languages to summary.json for cov. leaderboard view

View all activity

updated a bucket 3 days ago

commoncrawl/commoncrawl

updated a dataset 3 days ago

commoncrawl/statistics

Viewer • Updated 3 days ago • 631k • 603 • 26

in commoncrawl/commonlid-results 5 days ago

Add supported_languages to summary.json for cov. leaderboard view

#3 opened 5 days ago by

catherinearnett

authored 2 papers 14 days ago

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Paper • 2601.18026 • Published Jan 25

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Paper • 2605.09063 • Published 19 days ago • 79

in commoncrawl/commonlid-results 16 days ago

Add CommonLingua (PleIAs/CommonLingua) results on commonlid + commonlid_nano

#2 opened 16 days ago by

published a dataset 20 days ago

commoncrawl/commonlid-results

Preview • Updated 21 days ago • 808

updated a Space 20 days ago

CommonLID Leaderboard

Explore the CommonLID language identification leaderboard

updated a dataset 21 days ago

commoncrawl/commonlid-results

Preview • Updated 21 days ago • 808

in commoncrawl/commonlid-results 21 days ago

Refresh results from refactor

#1 opened 21 days ago by

published a Space 21 days ago

CommonLID Leaderboard

Explore the CommonLID language identification leaderboard

updated a bucket 29 days ago

commoncrawl/test-bucket

updated a dataset 30 days ago

commoncrawl/statistics

Viewer • Updated 3 days ago • 631k • 603 • 26

published a bucket about 1 month ago

commoncrawl/commoncrawl

authored 2 papers about 1 month ago

SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

Paper • 2512.11192 • Published Dec 12, 2025 • 1

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Paper • 2601.18026 • Published Jan 25

updated a dataset about 2 months ago

commoncrawl/citations

Viewer • Updated Apr 2 • 9.18k • 127 • 2

updated a dataset 2 months ago

commoncrawl/statistics

Viewer • Updated 3 days ago • 631k • 603 • 26

pjox

authored a paper 4 months ago

SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

Paper • 2512.11192 • Published Dec 12, 2025 • 1

catherinearnett

authored a paper 7 months ago

Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

Paper • 2506.01732 • Published Jun 2, 2025 • 6