Skip to content

Dataset · openbmb

Large Filtered En/Zh Web Corpus

openbmb/Ultra-FineWeb

1-10 billion English and Chinese texts after a quality check pipeline.

What's inside

A large English and Chinese text corpus filtered via an efficient quality check pipeline. Assembled from FineWeb and Chinese FineWeb; source data includes IndustryCorpus2, MiChao, WuDao, SkyPile, among others. Metadata indicates a volume of 1 to 10 billion examples.

What you can do with it

  • Pretrain language model on en/zh
  • Continued pretraining with clean web text
  • Evaluate data filtering pipelines
Tasks
Text generation
Languages
English, Chinese
Size
1B–10B rows
Downloads this month
40.1K
Likes
466

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("openbmb/Ultra-FineWeb")