Dataset · MMInstruction
Multimodal Preference Dataset with GPT-4V Annotations
MMInstruction/VLFeedback
80K image-instruction pairs with responses from 12 vision-language models evaluated by GPT-4V.
What's inside
It compiles 80,000 multimodal instructions (images and text) from various sources, testing different vision-language model (LVLM) skills. For each sample, 4 responses are generated from a pool of 12 models, with preference annotations performed by GPT-4V. The dataset size falls between 10,000 and 100,000 records.
What you can do with it
- RLHF or DPO fine-tuning of multimodal models (VLM)
- Training multimodal reward models to evaluate image-based responses
- Tasks
- Visual question answering
- Size
- 10K–100K rows
- Downloads this month
- 3K
- Likes
- 51
Models trained on it
- unum-cloud/uform-gen2-dpoImage → text 5K
- MMInstruction/SilkieText generation 74
- quiyver/intern-vl-2.5-rewardEmbeddings 11
- thaitea/laya-visionVisual question answering 0
- thaitea/laya-vision-smolvlm-256m-scoreVisual question answering 0
- thaitea/laya-vision-201mVisual question answering 0
- thaitea/laya-vision-microduck-kickVisual question answering 0
How to load it
from datasets import load_dataset
ds = load_dataset("MMInstruction/VLFeedback")