Skip to content

Dataset · VatsaDev

Collection of texts and tasks for training compact LLMs

VatsaDev/TinyText

A multi-task English text corpus of over 1 million entries for training small language models.

What's inside

The dataset is a single train.jsonl file ranging from 1 to 10 million English entries, associated with the NanoPhi project. It combines data across six domains: mathematics (Metamath, mammoth), code (Code Search Net), logic (Open-platypus), roleplay dialogues (PIPPA, RoleplayIO), synthetic textbooks (Tiny-text, Sciphi), and textbook QA (Orca-text, Tiny-webtext).

What you can do with it

  • Pre-training and fine-tuning compact language models similar to the Phi family
  • Training multi-task models in programming, logic, and mathematical problem-solving skills
Tasks
Question answering, Text generation
Languages
English
Size
1M–10M rows
Downloads this month
916
Likes
34

How to load it

from datasets import load_dataset

ds = load_dataset("VatsaDev/TinyText")