Skip to content

Dataset · ByteDance-Seed

Synthetic paraphrased FineWeb-Edu corpus

ByteDance-Seed/mga-fineweb-edu

Augmented text corpus by ByteDance for language model pre-training.

What's inside

A synthetic English-language pre-training dataset created by ByteDance-Seed researchers based on the SmolLM Corpus. Each source document was rewritten into 5 new variants considering different combinations of genres and target audiences, increasing the token volume by 3.9 times. The corpus size ranges from 100 million to 1 billion records.

What you can do with it

  • Pretraining and continued pretraining of large language models
  • Studying the impact of synthetic data augmentation on text generation quality
Tasks
Text generation
Languages
English
Size
100M–1B rows
Downloads this month
2.6K
Likes
44

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("ByteDance-Seed/mga-fineweb-edu")