SmolLM-Corpus
a dataset for training compact language models
2 min read
SmolLM-Corpus is a curated collection of high-quality educational and synthetic data created by the HuggingFace team to train small language models (LLMs). Inside the dataset are three main subsets: a massive array of synthetic textbooks and stories called Cosmopedia v2, high-quality Python code (Python-Edu), and the deduplicated educational web corpus FineWeb-Edu. This dataset is ideal for researchers and developers looking to train or fine-tune compact models from scratch.
Key features
- Cosmopedia v2: over 39 million synthetic textbooks, blog posts, and stories generated with Mixtral-8x7B-Instruct-v0.1.
- Python-Edu: filtered Python files with a high educational score, extracted from The Stack v2 repositories.
- FineWeb-Edu-Dedup: 220 billion tokens of deduplicated web pages of high educational quality.
- Ready to use: full integration with the datasets library for fast loading.
What you can do
- Train your own compact language model (LLM) from scratch on high-quality synthetic and text data.
- Build a specialized model for generating and analyzing Python code using the Python-Edu subset.
- Run pre-training experiments and benchmark small language model architectures.
Example
Loading the Cosmopedia v2 subset using the datasets library:
from datasets import load_dataset
ds = load_dataset("HuggingFaceTB/smollm-corpus", "cosmopedia-v2", split="train", num_proc=16)
print(ds[0])Read more
DetailsConsists of three key subsets: Cosmopedia v2, Python-Edu, and FineWeb-Edu (deduplicated).
Working with Python-Edu requires access to the Software Heritage S3 bucket.
Published by the HuggingFace team (Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, Leandro von Werra) in July 2024.








Comments