Dataset · OpenGVLab
Multimodal video and image instruction annotations
OpenGVLab/VideoChat2-IT
1.9M text annotations in JSON format for training models to answer questions about images and videos.
What's inside
The dataset contains 1.9M instructions and annotations in English in JSON format (media files must be downloaded separately per author instructions). Part of the image data is taken from the M3IT corpus and additionally cleaned of typos and punctuation errors.
What you can do with it
- Training vision-language models (VLMs) to answer photo and video questions
- SFT tuning of multimodal assistants for complex instruction execution
- Tasks
- Visual question answering, Question answering
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 674
- Likes
- 52
- Access
- on request on Hugging Face
Models trained on it
- DAMO-NLP-SG/VideoLLaMA2-7B-16FVisual question answering 131
- DAMO-NLP-SG/VideoLLaMA2-7BVisual question answering 93
- ermu2001/pllava-7bText generation 84
- DAMO-NLP-SG/VideoLLaMA2.1-7B-16FVideo text → text 69
- ermu2001/pllava-34bText generation 59
- ColorfulAI/videollamb-llava-1.5-7bVideo text → text 40
- Aliayub1995/VideoLLaMA2-7BVisual question answering 33
- DAMO-NLP-SG/VideoLLaMA2-8x7B-BaseVisual question answering 26
How to load it
from datasets import load_dataset
ds = load_dataset("OpenGVLab/VideoChat2-IT")