Dataset · google-research-datasets
Multilingual sentence pairs for paraphrase detection
google-research-datasets/paws-x
Over 320,000 sentence pairs across six languages for testing and training models in semantic understanding.
What's inside
The dataset from Google Research contains text pairs for paraphrase identification in German, Spanish, French, Chinese, Japanese, and Korean (English is also specified in the metadata). Inside are 23,659 human-translated validation pairs and 296,406 machine-translated training pairs from the PAWS-Wiki corpus. The dataset falls into the category of adversarial data where sentences look similar in words but may differ in meaning.
What you can do with it
- Training classifiers to accurately determine semantic similarity and paraphrases
- Cross-lingual evaluation of language model robustness to complex lexical traps
- Fine-tuning multilingual embeddings for search and text deduplication systems
- Tasks
- Text classification
- Languages
- German, English, Spanish, French, Japanese, Korean, Chinese
- Size
- 100K–1M rows
- Downloads this month
- 7.5K
- Likes
- 54
Models trained on it
- pkshatech/GLuCoSE-base-ja-v2Sentence similarity 5.8K
- blackmallangcow/nari-decision-4bEmbeddings 441
- pkshatech/RoSEtta-base-jaSentence similarity 351
- KeytechSRL/TernaryDecider-v1-Decision-LoRAText classification 124
- Andrew93110/sn92-gen-v6 82
- Andrew93110/sn92-gen-v5 52
- blackmallangcow/nari-decision-9bEmbeddings 47
- Guru0381/krino-2bText classification 37
How to load it
from datasets import load_dataset
ds = load_dataset("google-research-datasets/paws-x")