Dataset · nahco318
Diagnostic benchmark for VLM judge evaluation
nahco318/Towards-Reliable-VLM-Judges
Simulator scenes, model responses, and step-by-step evaluations to test the reliability of multimodal models as judges.
What's inside
A dataset accompanying a research paper on the reliability of multimodal large language models (VLM) when evaluating other models. It contains rendered observations from MiniGrid, MiniWorld, and Habitat-Sim/HSSD simulators, model responses, and evaluations for each question in English. The total size ranges from 1,000 to 10,000 records, combining images, tables, and text.
What you can do with it
- Evaluation and diagnosis of VLM models used as automated judges (LLM/VLM-as-a-Judge)
- Testing the robustness of multimodal models to changes in visual information presentation
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 5.7K
- Likes
- 2
How to load it
from datasets import load_dataset
ds = load_dataset("nahco318/Towards-Reliable-VLM-Judges")