Dataset · idoco
PopVQA benchmark for evaluating VLM factual knowledge via images
idoco/PopVQA
English image-question dataset for evaluating factual knowledge gaps in VLMs between text and photos.
What's inside
The dataset includes 1,000 to 10,000 English examples featuring images of popular entities and factual questions about them. It was created to investigate how much worse multimodal models answer questions about an object when it is shown in an image rather than described textually.
What you can do with it
- Evaluating factual accuracy of vision-language models (VLMs)
- Diagnosing the modality gap in recognizing known objects
- Tasks
- Image text → text
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 60.6K
- Likes
- 5
How to load it
from datasets import load_dataset
ds = load_dataset("idoco/PopVQA")