Skip to content

Dataset · unimelb-nlp

Multilingual NER on Wikipedia articles

unimelb-nlp/wikiann

176 languages, LOC/PER/ORG markup in IOB2 based on Wikipedia articles.

What's inside

WikiANN (PAN-X) is a multilingual dataset for NER. Wikipedia articles are tagged with LOC, PER, and ORG in the IOB2 format. The version corresponds to the balanced train/dev/test splits of Rahimi et al. (2019) and covers 176 out of 282 languages of the original corpus. Size ranges from 1M to 10M examples.

What you can do with it

  • Training a multilingual NER model
  • Testing token-classification on low-resource languages
  • Fine-tuning XLM-R or mBERT for entity extraction
Tasks
Token classification
Languages
Acehnese, Afrikaans, Albanian, Amharic, Aragonese, Old English, Arabic, Aramaic and 4 more
Size
1M–10M rows
Downloads this month
15.7K
Likes
126

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("unimelb-nlp/wikiann")