Skip to content

Dataset · allenai

Multilingual document corpus across 419 languages

allenai/MADLAD-400

A Common Crawl document dataset spanning 419 languages, audited and more heavily filtered than alternatives.

What's inside

This is a multilingual document corpus based on Common Crawl, covering 419 languages using all CommonCrawl snapshots up to August 1, 2022. According to the card, it is more multilingual, audited, and strictly filtered compared to similar datasets, with data organized at the document level. Metadata indicates a size greater than 1T.

What you can do with it

  • Pretraining multilingual LLMs
  • Fine-tuning and evaluation of low-resource language models
  • Researching Common Crawl filtering and auditing
Tasks
Text generation
Size
over 1T rows
Downloads this month
327.1K
Likes
185

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/MADLAD-400")