Skip to content
Datasets

SmolLM-Corpus

a dataset for training compact language models

2 min read

SmolLM-Corpus is a curated collection of high-quality educational and synthetic data created by the HuggingFace team to train small language models (LLMs). Inside the dataset are three main subsets: a massive array of synthetic textbooks and stories called Cosmopedia v2, high-quality Python code (Python-Edu), and the deduplicated educational web corpus FineWeb-Edu. This dataset is ideal for researchers and developers looking to train or fine-tune compact models from scratch.

Key features

  • Cosmopedia v2: over 39 million synthetic textbooks, blog posts, and stories generated with Mixtral-8x7B-Instruct-v0.1.
  • Python-Edu: filtered Python files with a high educational score, extracted from The Stack v2 repositories.
  • FineWeb-Edu-Dedup: 220 billion tokens of deduplicated web pages of high educational quality.
  • Ready to use: full integration with the datasets library for fast loading.

What you can do

  • Train your own compact language model (LLM) from scratch on high-quality synthetic and text data.
  • Build a specialized model for generating and analyzing Python code using the Python-Edu subset.
  • Run pre-training experiments and benchmark small language model architectures.

Example

Loading the Cosmopedia v2 subset using the datasets library:

from datasets import load_dataset

ds = load_dataset("HuggingFaceTB/smollm-corpus", "cosmopedia-v2", split="train", num_proc=16)
print(ds[0])
Read moreDetails
Consists of three key subsets: Cosmopedia v2, Python-Edu, and FineWeb-Edu (deduplicated).
Working with Python-Edu requires access to the Software Heritage S3 bucket.
Published by the HuggingFace team (Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, Leandro von Werra) in July 2024.
Discuss Telegram 524 70.5K

About the model

datasets/HuggingFaceTB/smollm-corpus
Author
HuggingFaceTB
Category
Datasets
Released
Posted

On NeyroLearn

Hugging Face

70.5K524

Ask AI about this model

Answered by Gemini 4 · the chat is kept only in this tab

Comments

Datasets

FineVision

FineVision