Dataset · SLPL
Largest cleaned Farsi text corpus
SLPL/naab
A massive Persian text corpus (130 GB, 15 billion words) for training language models from scratch and fine-tuning.
What's inside
Contains cleaned Farsi texts totaling about 130 GB. The dataset includes 250 million paragraphs and 15 billion words (10 to 100 million records in metadata). According to the authors, this is the largest ready-to-use open corpus for the Persian language.
What you can do with it
- Pre-training language models (LLMs) and encoders from scratch in Farsi
- Continual pre-training to adapt multilingual models to the Persian language
- Training tokenizers for Farsi texts
- Tasks
- Fill mask, Text generation
- Languages
- Persian
- Size
- 10M–100M rows
- Downloads this month
- 3.6K
- Likes
- 45
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("SLPL/naab")