Dataset · RekaAI
Vibe-Eval benchmark for multimodal chat models
RekaAI/VibeEval
A compact set of challenging vision-language tasks with reference answers in four languages.
What's inside
The dataset contains fewer than 1,000 examples for evaluating multimodal models capable of image-based dialogue. Each example includes an image, a user prompt, a reference answer, and a difficulty label (normal or hard). The data covers English, Polish, Chinese, and Japanese and features particularly challenging cases.
What you can do with it
- Evaluating multimodal chat model response quality on complex visual questions
- Comparing VLMs on multilingual tasks using reference answers
- Tasks
- Image → text, Image classification
- Languages
- English, Polish, Chinese, Japanese
- Size
- up to 1K rows
- Downloads this month
- 1.9K
- Likes
- 49
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("RekaAI/VibeEval")