Skip to content

Dataset · occiglot

Tokenizer benchmark on Wikipedia texts for 45 languages

occiglot/tokenizer-wiki-bench

Between 10 and 100 million pre-processed Wikipedia texts for evaluating tokenization efficiency across 45 languages.

What's inside

The dataset includes between 10 and 100 million pre-processed Wikipedia text snippets in 45 languages (including Afrikaans, Arabic, Bulgarian, Catalan, Czech, Danish, Slovak, and others). The data is designed to calculate tokenizer quality metrics such as fertility (average tokens per word) and split-word ratios.

What you can do with it

  • Compare the performance of ready-made LLM tokenizers across different languages
  • Evaluate custom vocabulary quality when training multilingual models
Languages
Afrikaans, Arabic, Bulgarian, Catalan, Czech, Danish, German, Greek and 4 more
Size
10M–100M rows
Downloads this month
19.8K
Likes
6

How to load it

from datasets import load_dataset

ds = load_dataset("occiglot/tokenizer-wiki-bench")