Dataset · commoncrawl
Web text language identification benchmark for 109 languages
commoncrawl/CommonLID
Manually annotated snippets of real web texts in 109 languages for evaluating language recognition algorithms.
What's inside
Dataset and benchmark from Common Crawl for the language identification (LID) task, containing from 100K to 1M lines. Inside are real web texts manually annotated by the community by writing language, covering 109 languages (at least 100 lines are presented for 78 of them).
What you can do with it
- Testing and comparing language identification classifiers on "dirty" web data
- Training filters to clean multilingual corpora during LLM pretraining
- Checking recognition quality of low-resource languages and dialects
- Tasks
- Text classification
- Languages
- Acehnese, acf, Tunisian Arabic, Afrikaans, Amharic, apd, Arabic, Arabic and 4 more
- Size
- 100K–1M rows
- Downloads this month
- 902
- Likes
- 59
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("commoncrawl/CommonLID")