Skip to content

Dataset · nampdn-ai

Synthetic textbooks for small LLMs

nampdn-ai/tiny-textbooks

100K-1M textbook-style English texts for training small language models.

What's inside

This is a synthetic corpus of textbook-like texts in English. Source texts are selected from falcon-refinedweb and minipile with an emphasis on diversity and quality at a small size. Synthesis was performed on 4x3090 Ti for 500 hours using fine-tuned Nous-Hermes-Llama2-13b. Metadata shows 100K-1M records; modalities are text and tabular data.

What you can do with it

  • Pretrain or SFT a small LLM
  • Experiments with synthetic training data in the spirit of phi-1.5
  • Checking educational text generation quality
Tasks
Text generation
Languages
English
Size
100K–1M rows
Downloads this month
318
Likes
190
Access
on request on Hugging Face

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("nampdn-ai/tiny-textbooks")