Skip to content

Dataset · HuggingFaceTB

Educational and synthetic text corpus for small LLMs

HuggingFaceTB/smollm-corpus

Large English corpus of educational and synthetic texts for pre-training small language models.

What's inside

Curated corpus of high-frequency educational and synthetic English data for training small language models. Volume is 100M-1B examples. Includes Cosmopedia v2: over 39 million synthetic textbooks, blogs, and stories. Format includes tabular and text data.

What you can do with it

  • Pre-training small language models
  • Fine-tuning LLMs on educational and synthetic texts
  • Experiments with Cosmopedia v2 for synthetic data
Languages
English
Size
100M–1B rows
Downloads this month
79.9K
Likes
526

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("HuggingFaceTB/smollm-corpus")