Common Crawl Foundation

non-profit

Verified

https://commoncrawl.org

AI & ML interests

Crawled data and metadata

Recent Activity

malteos updated a bucket about 11 hours ago

commoncrawl/commoncrawl

malteos authored a paper 2 days ago

Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data

malteos authored a paper 2 days ago

SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

View all activity

updated a bucket about 11 hours ago

commoncrawl/commoncrawl

authored 3 papers 2 days ago

Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data

Paper • 2507.00152 • Published Jun 30, 2025 • 1

SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

Paper • 2512.11192 • Published Dec 12, 2025 • 1

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Paper • 2601.18026 • Published Jan 25

updated a bucket 3 days ago

commoncrawl/warc2zip-examples

updated a bucket 3 days ago

commoncrawl/test-bucket

in commoncrawl/commonlid-results 4 days ago

Add GoogleTranslate-v3 results on commonlid_nano

#8 opened 4 days ago by

Rename GoogleTranslate to GoogleTranslate-v2

#7 opened 4 days ago by

updated 2 datasets 4 days ago

commoncrawl/host-index-testing-v2

Viewer • Updated 4 days ago • 9B • 7.72k

commoncrawl/statistics

Viewer • Updated 4 days ago • 647k • 1.14k • 30

in commoncrawl/host-index-testing-v2 5 days ago

Update README.md

#1 opened 5 days ago by

updated 2 Spaces 9 days ago

Web Graph Embeddings

What is actually inside llms.txt

Analyze llms.txt files for content insights

in commoncrawl/commonlid-results 10 days ago

Adding results from GoogleTranslate for commonlid_nano

#5 opened 10 days ago by

updated a bucket 11 days ago

commoncrawl/warc2zip-examples

updated a dataset 12 days ago

commoncrawl/llms.txt

Viewer • Updated 12 days ago • 598k • 214 • 2

published a dataset 12 days ago

commoncrawl/web-graph-embeddings

Viewer • Updated 12 days ago • 52.9M • 154 • 3

published a Space 12 days ago

Web Graph kNN Spot-check

Nearest neighbours of a host in the CC host-graph embedding

published a bucket 12 days ago

commoncrawl/web-graph-knn-index

updated a Space 12 days ago

Web Graph kNN Spot-check

Nearest neighbours of a host in the CC host-graph embedding