Skip to content

Dataset · nahco318

Diagnostic benchmark for VLM judge evaluation

nahco318/Towards-Reliable-VLM-Judges

Simulator scenes, model responses, and step-by-step evaluations to test the reliability of multimodal models as judges.

What's inside

A dataset accompanying a research paper on the reliability of multimodal large language models (VLM) when evaluating other models. It contains rendered observations from MiniGrid, MiniWorld, and Habitat-Sim/HSSD simulators, model responses, and evaluations for each question in English. The total size ranges from 1,000 to 10,000 records, combining images, tables, and text.

What you can do with it

  • Evaluation and diagnosis of VLM models used as automated judges (LLM/VLM-as-a-Judge)
  • Testing the robustness of multimodal models to changes in visual information presentation
Languages
English
Size
1K–10K rows
Downloads this month
5.7K
Likes
2

How to load it

from datasets import load_dataset

ds = load_dataset("nahco318/Towards-Reliable-VLM-Judges")