Skip to content

Dataset · idoco

PopVQA benchmark for evaluating VLM factual knowledge via images

idoco/PopVQA

English image-question dataset for evaluating factual knowledge gaps in VLMs between text and photos.

What's inside

The dataset includes 1,000 to 10,000 English examples featuring images of popular entities and factual questions about them. It was created to investigate how much worse multimodal models answer questions about an object when it is shown in an image rather than described textually.

What you can do with it

  • Evaluating factual accuracy of vision-language models (VLMs)
  • Diagnosing the modality gap in recognizing known objects
Tasks
Image text → text
Languages
English
Size
1K–10K rows
Downloads this month
60.6K
Likes
5

How to load it

from datasets import load_dataset

ds = load_dataset("idoco/PopVQA")