Dataset · VatsaDev
Collection of texts and tasks for training compact LLMs
VatsaDev/TinyText
A multi-task English text corpus of over 1 million entries for training small language models.
What's inside
The dataset is a single train.jsonl file ranging from 1 to 10 million English entries, associated with the NanoPhi project. It combines data across six domains: mathematics (Metamath, mammoth), code (Code Search Net), logic (Open-platypus), roleplay dialogues (PIPPA, RoleplayIO), synthetic textbooks (Tiny-text, Sciphi), and textbook QA (Orca-text, Tiny-webtext).
What you can do with it
- Pre-training and fine-tuning compact language models similar to the Phi family
- Training multi-task models in programming, logic, and mathematical problem-solving skills
- Tasks
- Question answering, Text generation
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 916
- Likes
- 34
How to load it
from datasets import load_dataset
ds = load_dataset("VatsaDev/TinyText")