Dataset · Zyphra
1.3T corpus for training language models
Zyphra/Zyda
Large English open dataset corpus with filtering and deduplication for LM training.
What's inside
Zyda is a 1.3T English language modeling dataset collected from open and high-quality sources. The authors combined sources and applied unified filtering and deduplication. According to the model card, the dataset performs well in ablations and is suitable for training language models.
What you can do with it
- Training or fine-tuning English LLMs
- Comparison with other open corpora in ablations
- Data preparation for language modeling
- Tasks
- Text generation
- Languages
- English
- Size
- 1B–10B rows
- Downloads this month
- 8.7K
- Likes
- 85
How to load it
from datasets import load_dataset
ds = load_dataset("Zyphra/Zyda")