Dataset · EleutherAI
Full Wikipedia articles at the document level
EleutherAI/wikitext_document_level
A modified version of the classic WikiText corpus, where texts are grouped into full pages.
What's inside
A text dataset with a volume of over 100 million tokens (from 10k to 100k records) extracted from verified Wikipedia featured and good articles. Unlike the original WikiText, which is split line by line, this version returns full articles at the document level. Distributed under a Creative Commons license.
What you can do with it
- Training and evaluation of language models on long context (document-level)
- Calculation of perplexity and classic text generation model benchmarking
- Size
- 10K–100K rows
- Downloads this month
- 106.1K
- Likes
- 28
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("EleutherAI/wikitext_document_level")