Skip to content

Dataset · jonathan-roberts1

ZeroBench: Complex visual benchmark for multimodal LLMs

jonathan-roberts1/zerobench

A set of under 1,000 complex multimodal questions with images for testing the limits of modern models.

What's inside

ZeroBench is a specialized benchmark for image-text-to-text tasks, positioned as extremely challenging for modern large multimodal models. The dataset contains fewer than 1,000 examples split into main questions and sub-questions, with an updated third version (v3) available in the main branch.

What you can do with it

  • Testing multimodal models (VLM) on complex visual analysis and reasoning tasks
  • Comparing state-of-the-art multimodal systems on questions where baseline benchmarks are already saturated
Tasks
Image text → text
Size
up to 1K rows
Downloads this month
50.2K
Likes
37
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("jonathan-roberts1/zerobench")