Dataset · occiglot
Tokenizer benchmark on Wikipedia texts for 45 languages
occiglot/tokenizer-wiki-bench
Between 10 and 100 million pre-processed Wikipedia texts for evaluating tokenization efficiency across 45 languages.
What's inside
The dataset includes between 10 and 100 million pre-processed Wikipedia text snippets in 45 languages (including Afrikaans, Arabic, Bulgarian, Catalan, Czech, Danish, Slovak, and others). The data is designed to calculate tokenizer quality metrics such as fertility (average tokens per word) and split-word ratios.
What you can do with it
- Compare the performance of ready-made LLM tokenizers across different languages
- Evaluate custom vocabulary quality when training multilingual models
- Languages
- Afrikaans, Arabic, Bulgarian, Catalan, Czech, Danish, German, Greek and 4 more
- Size
- 10M–100M rows
- Downloads this month
- 19.8K
- Likes
- 6
How to load it
from datasets import load_dataset
ds = load_dataset("occiglot/tokenizer-wiki-bench")