Dataset · HuggingFaceTB
Educational and synthetic text corpus for small LLMs
HuggingFaceTB/smollm-corpus
Large English corpus of educational and synthetic texts for pre-training small language models.
What's inside
Curated corpus of high-frequency educational and synthetic English data for training small language models. Volume is 100M-1B examples. Includes Cosmopedia v2: over 39 million synthetic textbooks, blogs, and stories. Format includes tabular and text data.
What you can do with it
- Pre-training small language models
- Fine-tuning LLMs on educational and synthetic texts
- Experiments with Cosmopedia v2 for synthetic data
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 79.9K
- Likes
- 526
Models trained on it
- HuggingFaceTB/SmolLM-135MText generation 140K
- HuggingFaceTB/SmolLM-1.7BText generation 25.7K
- dmis-lab/UltraBERT-150MFill mask 12.6K
- dmis-lab/UltraBERT-380MFill mask 9.8K
- QuantFactory/SmolLM-135M-GGUF 8.3K
- HuggingFaceTB/SmolLM-360MText generation 7K
- MinimaLabs/KLM-1.6Text generation 3.1K
- MatterWaveAI/grt-30mText generation 2.7K
How to load it
from datasets import load_dataset
ds = load_dataset("HuggingFaceTB/smollm-corpus")