Dataset · Skylion007
Open WebText replica for GPT-2 training
Skylion007/openwebtext
Open English replica of OpenAI's WebText used to train GPT-2.
What's inside
OpenWebText is an open replica of OpenAI's WebText dataset used to train GPT-2. Created by Aaron Gokaslan and Vanya Cohen from Brown University. Contains English texts; size ranges from 1M to 10M examples. Text generation and fill-mask tasks are specified.
What you can do with it
- Pretraining small language models
- Fine-tuning for text generation
- Testing fill-mask models
- Tasks
- Text generation, Fill mask
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 54.8K
- Likes
- 552
Models trained on it
- erwanf/gpt2-miniText generation 409K
- kuleshov-group/mdlm-owtFill mask 26.8K
- Continuous-Rivals-Discrete/langflow-owtText generation 1.8K
- Felladrin/Minueza-32M-BaseText generation 1.2K
- dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm2-pmx16-sr2-fobText generation 931
- dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2-fobText generation 919
- dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2Text generation 911
- dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm2-pmx16-sr2Text generation 894
How to load it
from datasets import load_dataset
ds = load_dataset("Skylion007/openwebtext")