Skip to content

Dataset · commoncrawl

Web text language identification benchmark for 109 languages

commoncrawl/CommonLID

Manually annotated snippets of real web texts in 109 languages for evaluating language recognition algorithms.

What's inside

Dataset and benchmark from Common Crawl for the language identification (LID) task, containing from 100K to 1M lines. Inside are real web texts manually annotated by the community by writing language, covering 109 languages (at least 100 lines are presented for 78 of them).

What you can do with it

  • Testing and comparing language identification classifiers on "dirty" web data
  • Training filters to clean multilingual corpora during LLM pretraining
  • Checking recognition quality of low-resource languages and dialects
Tasks
Text classification
Languages
Acehnese, acf, Tunisian Arabic, Afrikaans, Amharic, apd, Arabic, Arabic and 4 more
Size
100K–1M rows
Downloads this month
902
Likes
59
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("commoncrawl/CommonLID")