Skip to content

Dataset · yifanzhang114

Multimodal Benchmark for Evaluating VLM Models

yifanzhang114/MME-RealWorld

A challenging test set for checking visual understanding and visual question answering.

What's inside

A large-scale English-language benchmark for multimodal models, supporting visual-question-answering and multiple-choice tasks. It contains between 100 billion and 1 trillion records (according to metadata) and has a lightweight lite version with 50 examples per task for quick evaluation. The set is integrated into popular testing frameworks VLMEvalKit and Lmms-eval, and even top models like LLaVA-OV score less than 55% accuracy on it.

What you can do with it

  • Evaluating the quality of multimodal models (VLMs) on complex visual tasks
  • Comparing models in a single environment via VLMEvalKit and Lmms-eval
  • Fast inference benchmarking using the lightweight lite version
Tasks
Multiple choice, Question answering, Visual question answering
Languages
English
Size
100B–1T rows
Downloads this month
5.3K
Likes
27

How to load it

from datasets import load_dataset

ds = load_dataset("yifanzhang114/MME-RealWorld")