Skip to content

Dataset · Battam

Open data index for all model training stages

Battam/training-data-foundation

A catalog of links, hashes, and licenses for real public data for pre-training, mid-training, and post-training.

What's inside

The dataset is a public index from the Training Data Foundation project containing between 10,000 and 100,000 records in tabular and text formats. For each file, its address, size, checksum hashes, and license are provided, and downloading the required sets (e.g., tables) is done with a single command line. The materials focus on English and Chinese and cover all stages of neural network training.

What you can do with it

  • Collecting and automating downloads of license-clean corpora for pre-training and mid-training
  • Building data splits for post-training tasks with file integrity verification via hashes
Size
10K–100K rows
Downloads this month
61K
Likes
3

How to load it

from datasets import load_dataset

ds = load_dataset("Battam/training-data-foundation")