Skip to content

Dataset · openbmb

Cleaned pre-training texts by UltraX

openbmb/UltraX-Preview

Large English corpus edited by the UltraX framework for language model pre-training.

What's inside

UltraX is a function-calling framework that adaptively generates and executes editing functions for point-wise cleaning of large-scale data. The dataset consists of English pre-training texts, with metadata indicating a size of 100M to 1B items. The dataset card links to the paper, code, and models, though the data format is unspecified.

What you can do with it

  • Pre-training a small language model on a cleaned corpus
  • Comparing training quality between cleaned and raw data
  • Configuring a text-cleaning pipeline via function calling
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
6.6K
Likes
302

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("openbmb/UltraX-Preview")