Dataset · instruction-pretrain
200M instruction-response pairs for LLM pre-training
instruction-pretrain/general-instruction-augmented-corpora
Massive English corpus of 200M instruction-response pairs across 40+ task categories for multi-task model pre-training.
What's inside
Contains 200 million English instruction-response pairs spanning over 40 diverse task categories, including text classification, question answering over text, and tabular reasoning. Created by scalably augmenting raw text corpora with instructions to research the Instruction Pre-Training method. Published alongside an EMNLP 2024 conference paper.
What you can do with it
- Multi-task instruction pre-training of base language models
- Fine-tuning models for table reasoning, question answering, and zero-shot classification
- Tasks
- Text classification, Table question answering, Question answering, Zero shot classification
- Languages
- English
- Downloads this month
- 20.1K
- Likes
- 31
Models trained on it
- mradermacher/Apertus-EstLLM-8B-1125-i1-GGUF 1.3K
- mradermacher/Llama-3.1-EstLLM-8B-0525-i1-GGUF 646
- mradermacher/InstructLM-500M-i1-GGUF 587
- mradermacher/InstructLM-1.3B-i1-GGUF 379
- instruction-pretrain/InstructLM-500MText generation 280
- mradermacher/Apertus-EstLLM-8B-1125-GGUF 261
- instruction-pretrain/InstructLM-1.3BText generation 251
- mradermacher/InstructLM-1.3B-GGUF 219
How to load it
from datasets import load_dataset
ds = load_dataset("instruction-pretrain/general-instruction-augmented-corpora")