Dataset · Tree-AI-lab
Multilingual corpus with rare African languages
Tree-AI-lab/smollm-corpus-multilingue
A text corpus of 10 to 100M records for pre-training models in English, French, and West African languages.
What's inside
Contains texts in five languages: English, French, Baoulé, Dioula, and Nushi, with a total volume ranging from 10 to 100 million records. Created by Tree AI Lab as part of the BABA project based on HuggingFaceTB/smollm-corpus. The base data is supplemented with math and deductive texts from finemath and OpenThoughts-114k, as well as large corpora in rare African languages.
What you can do with it
- Pre-training compact language models with support for African languages
- Continual pre-training of existing LLMs in Baoulé, Dioula, and Nushi
- Improving logic and mathematics skills in multilingual models
- Tasks
- Text generation
- Languages
- English, French, bci, Dyula, nch
- Size
- 10M–100M rows
- Downloads this month
- 32.4K
- Likes
- 2
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("Tree-AI-lab/smollm-corpus-multilingue")