Skip to content

Dataset · allenai

Long and detailed image descriptions for VLMs

allenai/pixmo-cap

Hundreds of thousands of images with detailed texts averaging 200 words and original speech transcripts.

What's inside

The dataset contains between 100 thousand and 1 million pairs of images and highly detailed text descriptions (averaging about 200 words). To create it, annotators voiced everything they saw in the image for 60-90 seconds, and then a Claude model turned the transcription into coherent text. The audio transcripts themselves are also included in the dataset. The set is part of the PixMo collection and was used in training the Molmo family of multimodal models.

What you can do with it

  • Pre-training and fine-tuning (SFT) multimodal models (VLMs) for generating detailed descriptions
  • Reducing hallucinations in vision-language models through training on detailed annotations
Tasks
Image → text
Size
100K–1M rows
Downloads this month
1.5K
Likes
49

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("allenai/pixmo-cap")