Skip to content

Dataset · mit-han-lab

The Pile validation split backup

mit-han-lab/pile-val-backup

A backup of the Pile text corpus validation split for language model evaluation.

What's inside

Contains texts from the validation split of the Pile corpus (originally file val.jsonl.zst from the-eye.eu). The dataset size is between 100,000 and 1 million entries. The card notes the requirement to follow the original dataset license.

What you can do with it

  • Calculating perplexity and evaluating language model quality
  • Calibration during LLM quantization and pruning
Size
100K–1M rows
Downloads this month
19.9K
Likes
27

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("mit-han-lab/pile-val-backup")