Dataset · papluca
Text language identification
papluca/language-identification
90k text fragments with language labels for classification.
What's inside
The dataset contains 90k examples: text fragments and corresponding language labels. Collected from Multilingual Amazon Reviews Corpus, XNLI, and STSb Multi MT. Suitable for training a multi-class language identification model; languages in metadata are ar, bg, de, el, en, es.
What you can do with it
- Training a language classifier
- Filtering multilingual texts
- Testing language identification pipelines
- Tasks
- Text classification
- Languages
- Arabic, Bulgarian, German, Greek, English, Spanish, French, Hindi and 4 more
- Size
- 10K–100K rows
- Downloads this month
- 3.4K
- Likes
- 74
Models trained on it
- papluca/xlm-roberta-base-language-detectionText classification 104.8K
- StrandsAgents/strands-decider-2B-hobson-v21Text classification 15.6K
- tasksource/deberta-small-long-nliZero shot classification 9.3K
- sileod/deberta-v3-base-tasksource-nliZero shot classification 8K
- StrandsAgents/strands-decider-2B-hobson-v19Text classification 7K
- dominguesm/xlm-roberta-base-lora-language-detection 2.2K
- sileod/deberta-v3-large-tasksource-nliZero shot classification 1.8K
- tasksource/deberta-base-long-nliZero shot classification 1.4K
How to load it
from datasets import load_dataset
ds = load_dataset("papluca/language-identification")