Dataset · yifanzhang114
Multimodal Benchmark for Evaluating VLM Models
yifanzhang114/MME-RealWorld
A challenging test set for checking visual understanding and visual question answering.
What's inside
A large-scale English-language benchmark for multimodal models, supporting visual-question-answering and multiple-choice tasks. It contains between 100 billion and 1 trillion records (according to metadata) and has a lightweight lite version with 50 examples per task for quick evaluation. The set is integrated into popular testing frameworks VLMEvalKit and Lmms-eval, and even top models like LLaVA-OV score less than 55% accuracy on it.
What you can do with it
- Evaluating the quality of multimodal models (VLMs) on complex visual tasks
- Comparing models in a single environment via VLMEvalKit and Lmms-eval
- Fast inference benchmarking using the lightweight lite version
- Tasks
- Multiple choice, Question answering, Visual question answering
- Languages
- English
- Size
- 100B–1T rows
- Downloads this month
- 5.3K
- Likes
- 27
How to load it
from datasets import load_dataset
ds = load_dataset("yifanzhang114/MME-RealWorld")