Skip to content

Dataset · anisoleai

Tokenized FineWeb corpus with 4+ trillion tokens

anisoleai/fineweb-tokenized

Pre-tokenized version of the giant English FineWeb web corpus in parquet format.

What's inside

A large English dataset of over 4 trillion tokens based on the well-known FineWeb corpus (full processing of all ~15 trillion tokens is still ongoing). Texts are pre-processed through the AnisoleAI BPE tokenizer with a 52,022-token vocabulary and packed into compact parquet shards with uint16 data type.

What you can do with it

  • Pre-training language models from scratch without tokenization overhead
  • Testing data loading pipeline throughput on massive volumes
Tasks
Text generation
Languages
English
Size
over 1T rows
Downloads this month
127.9K
Likes
40

How to load it

from datasets import load_dataset

ds = load_dataset("anisoleai/fineweb-tokenized")