Dataset · allenai
Multilingual document corpus across 419 languages
allenai/MADLAD-400
A Common Crawl document dataset spanning 419 languages, audited and more heavily filtered than alternatives.
What's inside
This is a multilingual document corpus based on Common Crawl, covering 419 languages using all CommonCrawl snapshots up to August 1, 2022. According to the card, it is more multilingual, audited, and strictly filtered compared to similar datasets, with data organized at the document level. Metadata indicates a size greater than 1T.
What you can do with it
- Pretraining multilingual LLMs
- Fine-tuning and evaluation of low-resource language models
- Researching Common Crawl filtering and auditing
- Tasks
- Text generation
- Size
- over 1T rows
- Downloads this month
- 327.1K
- Likes
- 185
Models trained on it
- google/madlad400-3b-mtTranslation 194.1K
- mtsdurica/madlad400-7b-mt-Q4_K_M-GGUFTranslation 4.9K
- Nextcloud-AI/madlad400-3b-mt-ct2-int8Translation 4.9K
- mradermacher/TildeOpen-30B-Instruct-LV-i1-GGUF 4.8K
- google/madlad400-10b-mtTranslation 3.4K
- jbochi/madlad400-3b-mtTranslation 3.3K
- TildeAI/TildeOpen-30bText generation 2.3K
- google/madlad400-7b-mtTranslation 2K
How to load it
from datasets import load_dataset
ds = load_dataset("allenai/MADLAD-400")