Skip to content

Dataset · AletheiaAING

Text quality assessment across 71 African languages

AletheiaAING/Data-Quality

A set of seven configurations for training text quality and naturalness classifiers in 71 African languages.

What's inside

The dataset combines seven subsets with different schemas for evaluating text quality in African languages without reference translation. The main config contains 493,831 human-written lines of text with known types and degrees of artificial distortions, while other configs contain tens of thousands of model-generated texts from prompts across 71 languages (including Afrikaans, Amharic, Arabic, and others).

What you can do with it

  • Training classifiers to filter out low-quality and machine-corrupted data during corpus collection
  • Building text generation quality assessment models (reward models) for low-resource African languages
  • Detecting synthetic texts across 71 African languages
Tasks
Text classification
Languages
Afrikaans, Akan, Amharic, Arabic, Bambara, Bemba, Bini, dag and 4 more
Size
100K–1M rows
Downloads this month
42K
Likes
0

How to load it

from datasets import load_dataset

ds = load_dataset("AletheiaAING/Data-Quality")