Dataset · Battam
Open data index for all model training stages
Battam/training-data-foundation
A catalog of links, hashes, and licenses for real public data for pre-training, mid-training, and post-training.
What's inside
The dataset is a public index from the Training Data Foundation project containing between 10,000 and 100,000 records in tabular and text formats. For each file, its address, size, checksum hashes, and license are provided, and downloading the required sets (e.g., tables) is done with a single command line. The materials focus on English and Chinese and cover all stages of neural network training.
What you can do with it
- Collecting and automating downloads of license-clean corpora for pre-training and mid-training
- Building data splits for post-training tasks with file integrity verification via hashes
- Size
- 10K–100K rows
- Downloads this month
- 61K
- Likes
- 3
How to load it
from datasets import load_dataset
ds = load_dataset("Battam/training-data-foundation")