Dataset · jhu-clsp
Text mixture for pre-training the Ettin model family
jhu-clsp/ettin-pretraining-data
Large corpus of 1.7 trillion English tokens in MDS format for language model pre-training.
What's inside
The collection represents the first of three data mixture phases in English used to train Ettin family encoders and decoders. It is built on cleaned web data, including DCLM (837.2 billion tokens, or 49.1% of the volume) and other web sources. Files are packaged in MDS format for direct compatibility with the Composer framework and ModernBERT repository.
What you can do with it
- Pre-training language models from scratch (encoders and decoders)
- Fine-tuning ModernBERT architecture models on high-quality web texts
- Tasks
- Text generation, Fill mask, Text classification
- Languages
- English
- Downloads this month
- 70.2K
- Likes
- 10
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("jhu-clsp/ettin-pretraining-data")