Skip to content

Dataset · HPLT

Cleaned HPLT 2.0 web corpus in 191 languages

HPLT/HPLT2.0_cleaned

Billions of cleaned web documents from Internet Archive and Common Crawl in 191 world languages.

What's inside

This is a cleaned version of a large-scale multilingual web document corpus collected by the HPLT project primarily from Internet Archive and Common Crawl archives. The volume ranges from 1 to 10 billion entries in 191 languages, including rare and low-resource ones. The authors note that version 2.0 is already outdated and recommend upgrading to the HPLT 3.0 release.

What you can do with it

  • Pre-training multilingual language models and tokenizers
  • Gathering text data for low-resource languages
  • Training fill-mask and text generation models
Tasks
Fill mask, Text generation
Languages
Acehnese, Afrikaans, Albanian, Amharic, Arabic, Assamese, Asturian, Awadhi and 4 more
Size
1B–10B rows
Downloads this month
119.3K
Likes
55

How to load it

from datasets import load_dataset

ds = load_dataset("HPLT/HPLT2.0_cleaned")
Cleaned HPLT 2.0 web corpus in 191 languages: dataset HPLT/HPLT2.0_cleaned | NeyroLearn