Dataset · AletheiaAING
Text quality assessment across 71 African languages
AletheiaAING/Data-Quality
A set of seven configurations for training text quality and naturalness classifiers in 71 African languages.
What's inside
The dataset combines seven subsets with different schemas for evaluating text quality in African languages without reference translation. The main config contains 493,831 human-written lines of text with known types and degrees of artificial distortions, while other configs contain tens of thousands of model-generated texts from prompts across 71 languages (including Afrikaans, Amharic, Arabic, and others).
What you can do with it
- Training classifiers to filter out low-quality and machine-corrupted data during corpus collection
- Building text generation quality assessment models (reward models) for low-resource African languages
- Detecting synthetic texts across 71 African languages
- Tasks
- Text classification
- Languages
- Afrikaans, Akan, Amharic, Arabic, Bambara, Bemba, Bini, dag and 4 more
- Size
- 100K–1M rows
- Downloads this month
- 42K
- Likes
- 0
How to load it
from datasets import load_dataset
ds = load_dataset("AletheiaAING/Data-Quality")