Skip to content

Dataset · emily9589

PG-19 long context book corpus

emily9589/sn38-r18-2011-raw-pg19

A copy of the PG-19 benchmark based on Project Gutenberg books for training and evaluating language models on long texts.

What's inside

The repository contains the PG-19 text corpus (volume ranging from 100M to 1B entries) gathered from Project Gutenberg library books published before 1919, along with metadata on titles and publication dates. Documents in the set are on average 20 times longer than in the WikiText benchmark, and the total volume is more than double that of Billion Word.

What you can do with it

  • Evaluating the ability of language models to maintain ultra-long context (long-range language modeling)
  • Pre-training or fine-tuning text models on classical English literature
Size
100M–1B rows
Downloads this month
31.5K
Likes
0
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("emily9589/sn38-r18-2011-raw-pg19")