Dataset · ByteDance-Seed
Synthetic paraphrased FineWeb-Edu corpus
ByteDance-Seed/mga-fineweb-edu
Augmented text corpus by ByteDance for language model pre-training.
What's inside
A synthetic English-language pre-training dataset created by ByteDance-Seed researchers based on the SmolLM Corpus. Each source document was rewritten into 5 new variants considering different combinations of genres and target audiences, increasing the token volume by 3.9 times. The corpus size ranges from 100 million to 1 billion records.
What you can do with it
- Pretraining and continued pretraining of large language models
- Studying the impact of synthetic data augmentation on text generation quality
- Tasks
- Text generation
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 2.6K
- Likes
- 44
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("ByteDance-Seed/mga-fineweb-edu")