Dataset · anisoleai
Tokenized FineWeb corpus with 4+ trillion tokens
anisoleai/fineweb-tokenized
Pre-tokenized version of the giant English FineWeb web corpus in parquet format.
What's inside
A large English dataset of over 4 trillion tokens based on the well-known FineWeb corpus (full processing of all ~15 trillion tokens is still ongoing). Texts are pre-processed through the AnisoleAI BPE tokenizer with a 52,022-token vocabulary and packed into compact parquet shards with uint16 data type.
What you can do with it
- Pre-training language models from scratch without tokenization overhead
- Testing data loading pipeline throughput on massive volumes
- Tasks
- Text generation
- Languages
- English
- Size
- over 1T rows
- Downloads this month
- 127.9K
- Likes
- 40
How to load it
from datasets import load_dataset
ds = load_dataset("anisoleai/fineweb-tokenized")