Dataset · jonathan-roberts1
ZeroBench: Complex visual benchmark for multimodal LLMs
jonathan-roberts1/zerobench
A set of under 1,000 complex multimodal questions with images for testing the limits of modern models.
What's inside
ZeroBench is a specialized benchmark for image-text-to-text tasks, positioned as extremely challenging for modern large multimodal models. The dataset contains fewer than 1,000 examples split into main questions and sub-questions, with an updated third version (v3) available in the main branch.
What you can do with it
- Testing multimodal models (VLM) on complex visual analysis and reasoning tasks
- Comparing state-of-the-art multimodal systems on questions where baseline benchmarks are already saturated
- Tasks
- Image text → text
- Size
- up to 1K rows
- Downloads this month
- 50.2K
- Likes
- 37
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("jonathan-roberts1/zerobench")