Skip to content

Dataset · Salesforce

WikiText: Wikipedia texts for language modeling

Salesforce/wikitext

Corpus of verified English Wikipedia articles for training and evaluating language models.

What's inside

The dataset contains over 100M tokens from featured and good English Wikipedia articles. These are English texts collected from Wikipedia and available under the CC BY-SA license. The WikiText-2 and WikiText-103 versions are noticeably larger than the preprocessed Penn Treebank. The format is text data for text-generation and fill-mask tasks.

What you can do with it

  • Train a GPT-like model for language modeling
  • Test English fill-mask
  • Evaluate language model perplexity
Tasks
Text generation, Fill mask
Languages
English
Size
1M–10M rows
Downloads this month
2.5M
Likes
870

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("Salesforce/wikitext")