Skip to content

Dataset · Skylion007

Open WebText replica for GPT-2 training

Skylion007/openwebtext

Open English replica of OpenAI's WebText used to train GPT-2.

What's inside

OpenWebText is an open replica of OpenAI's WebText dataset used to train GPT-2. Created by Aaron Gokaslan and Vanya Cohen from Brown University. Contains English texts; size ranges from 1M to 10M examples. Text generation and fill-mask tasks are specified.

What you can do with it

  • Pretraining small language models
  • Fine-tuning for text generation
  • Testing fill-mask models
Tasks
Text generation, Fill mask
Languages
English
Size
1M–10M rows
Downloads this month
54.8K
Likes
552

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("Skylion007/openwebtext")