Skip to content

Dataset · RekaAI

Vibe-Eval benchmark for multimodal chat models

RekaAI/VibeEval

A compact set of challenging vision-language tasks with reference answers in four languages.

What's inside

The dataset contains fewer than 1,000 examples for evaluating multimodal models capable of image-based dialogue. Each example includes an image, a user prompt, a reference answer, and a difficulty label (normal or hard). The data covers English, Polish, Chinese, and Japanese and features particularly challenging cases.

What you can do with it

  • Evaluating multimodal chat model response quality on complex visual questions
  • Comparing VLMs on multilingual tasks using reference answers
Tasks
Image → text, Image classification
Languages
English, Polish, Chinese, Japanese
Size
up to 1K rows
Downloads this month
1.9K
Likes
49

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("RekaAI/VibeEval")