Skip to content

Dataset · marin-community

Marin pretraining token statistics

marin-community/token-counts

Token count information for all datasets used in pretraining Marin models.

What's inside

Contains text data describing dataset IDs, post-tokenization token counts, content domains (web, code, math, academic texts, books), and a synthetic data flag. The dataset size is under 1K entries.

What you can do with it

  • Analyzing token composition and distribution in training corpora
  • Researching data domains for LLM pretraining
Tasks
Text generation
Size
up to 1K rows
Downloads this month
771
Likes
3

How to load it

from datasets import load_dataset

ds = load_dataset("marin-community/token-counts")