Skip to content

Dataset · Zyphra

1.3T corpus for training language models

Zyphra/Zyda

Large English open dataset corpus with filtering and deduplication for LM training.

What's inside

Zyda is a 1.3T English language modeling dataset collected from open and high-quality sources. The authors combined sources and applied unified filtering and deduplication. According to the model card, the dataset performs well in ablations and is suitable for training language models.

What you can do with it

  • Training or fine-tuning English LLMs
  • Comparison with other open corpora in ablations
  • Data preparation for language modeling
Tasks
Text generation
Languages
English
Size
1B–10B rows
Downloads this month
8.7K
Likes
85

How to load it

from datasets import load_dataset

ds = load_dataset("Zyphra/Zyda")