Skip to content

Dataset · google-research-datasets

Multilingual sentence pairs for paraphrase detection

google-research-datasets/paws-x

Over 320,000 sentence pairs across six languages for testing and training models in semantic understanding.

What's inside

The dataset from Google Research contains text pairs for paraphrase identification in German, Spanish, French, Chinese, Japanese, and Korean (English is also specified in the metadata). Inside are 23,659 human-translated validation pairs and 296,406 machine-translated training pairs from the PAWS-Wiki corpus. The dataset falls into the category of adversarial data where sentences look similar in words but may differ in meaning.

What you can do with it

  • Training classifiers to accurately determine semantic similarity and paraphrases
  • Cross-lingual evaluation of language model robustness to complex lexical traps
  • Fine-tuning multilingual embeddings for search and text deduplication systems
Tasks
Text classification
Languages
German, English, Spanish, French, Japanese, Korean, Chinese
Size
100K–1M rows
Downloads this month
7.5K
Likes
54

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("google-research-datasets/paws-x")