Dataset · saheedniyi
Four-language Nigerian web text corpus
saheedniyi/naijaweb
Over 270k web documents (~230M tokens) in English, Yoruba, Hausa, and Igbo.
What's inside
A text corpus in parquet format containing over 270k documents (about 230M GPT-2 tokens) gathered from popular Nigerian web pages. Materials are presented in English, Yoruba, Hausa, and Igbo, accompanied by metadata on language, domain, token count, and text quality score. The dataset was created similarly to FineWeb to convey Nigerian linguistic and cultural context.
What you can do with it
- Pre-training and language adaptation of LLMs for low-resource African languages
- Filtering high-quality educational texts using the built-in int_score metric
- Tasks
- Text generation
- Languages
- English, Yoruba, Hausa, Igbo
- Size
- 100K–1M rows
- Downloads this month
- 70
- Likes
- 39
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("saheedniyi/naijaweb")