Skip to content

Dataset · MMInstruction

Multimodal Preference Dataset with GPT-4V Annotations

MMInstruction/VLFeedback

80K image-instruction pairs with responses from 12 vision-language models evaluated by GPT-4V.

What's inside

It compiles 80,000 multimodal instructions (images and text) from various sources, testing different vision-language model (LVLM) skills. For each sample, 4 responses are generated from a pool of 12 models, with preference annotations performed by GPT-4V. The dataset size falls between 10,000 and 100,000 records.

What you can do with it

  • RLHF or DPO fine-tuning of multimodal models (VLM)
  • Training multimodal reward models to evaluate image-based responses
Tasks
Visual question answering
Size
10K–100K rows
Downloads this month
3K
Likes
51

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("MMInstruction/VLFeedback")