Skip to content

Dataset · jhu-clsp

Text mixture for pre-training the Ettin model family

jhu-clsp/ettin-pretraining-data

Large corpus of 1.7 trillion English tokens in MDS format for language model pre-training.

What's inside

The collection represents the first of three data mixture phases in English used to train Ettin family encoders and decoders. It is built on cleaned web data, including DCLM (837.2 billion tokens, or 49.1% of the volume) and other web sources. Files are packaged in MDS format for direct compatibility with the Composer framework and ModernBERT repository.

What you can do with it

  • Pre-training language models from scratch (encoders and decoders)
  • Fine-tuning ModernBERT architecture models on high-quality web texts
Tasks
Text generation, Fill mask, Text classification
Languages
English
Downloads this month
70.2K
Likes
10

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("jhu-clsp/ettin-pretraining-data")