Dataset · marin-community
Marin pretraining token statistics
marin-community/token-counts
Token count information for all datasets used in pretraining Marin models.
What's inside
Contains text data describing dataset IDs, post-tokenization token counts, content domains (web, code, math, academic texts, books), and a synthetic data flag. The dataset size is under 1K entries.
What you can do with it
- Analyzing token composition and distribution in training corpora
- Researching data domains for LLM pretraining
- Tasks
- Text generation
- Size
- up to 1K rows
- Downloads this month
- 771
- Likes
- 3
How to load it
from datasets import load_dataset
ds = load_dataset("marin-community/token-counts")