Skip to content

Dataset · instruction-pretrain

200M instruction-response pairs for LLM pre-training

instruction-pretrain/general-instruction-augmented-corpora

Massive English corpus of 200M instruction-response pairs across 40+ task categories for multi-task model pre-training.

What's inside

Contains 200 million English instruction-response pairs spanning over 40 diverse task categories, including text classification, question answering over text, and tabular reasoning. Created by scalably augmenting raw text corpora with instructions to research the Instruction Pre-Training method. Published alongside an EMNLP 2024 conference paper.

What you can do with it

  • Multi-task instruction pre-training of base language models
  • Fine-tuning models for table reasoning, question answering, and zero-shot classification
Tasks
Text classification, Table question answering, Question answering, Zero shot classification
Languages
English
Downloads this month
20.1K
Likes
31

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("instruction-pretrain/general-instruction-augmented-corpora")