Dataset · HPLT
Cleaned HPLT 2.0 web corpus in 191 languages
HPLT/HPLT2.0_cleaned
Billions of cleaned web documents from Internet Archive and Common Crawl in 191 world languages.
What's inside
This is a cleaned version of a large-scale multilingual web document corpus collected by the HPLT project primarily from Internet Archive and Common Crawl archives. The volume ranges from 1 to 10 billion entries in 191 languages, including rare and low-resource ones. The authors note that version 2.0 is already outdated and recommend upgrading to the HPLT 3.0 release.
What you can do with it
- Pre-training multilingual language models and tokenizers
- Gathering text data for low-resource languages
- Training fill-mask and text generation models
- Tasks
- Fill mask, Text generation
- Languages
- Acehnese, Afrikaans, Albanian, Amharic, Arabic, Assamese, Asturian, Awadhi and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 119.3K
- Likes
- 55
How to load it
from datasets import load_dataset
ds = load_dataset("HPLT/HPLT2.0_cleaned")