Skip to content

Dataset · papluca

Text language identification

papluca/language-identification

90k text fragments with language labels for classification.

What's inside

The dataset contains 90k examples: text fragments and corresponding language labels. Collected from Multilingual Amazon Reviews Corpus, XNLI, and STSb Multi MT. Suitable for training a multi-class language identification model; languages in metadata are ar, bg, de, el, en, es.

What you can do with it

  • Training a language classifier
  • Filtering multilingual texts
  • Testing language identification pipelines
Tasks
Text classification
Languages
Arabic, Bulgarian, German, Greek, English, Spanish, French, Hindi and 4 more
Size
10K–100K rows
Downloads this month
3.4K
Likes
74

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("papluca/language-identification")