Skip to content

Dataset · mlfoundations

DataComp-VLM multilingual image-text pool

mlfoundations/dcvlm_pool_large

Huge raw pool of nearly 2 billion image-text pairs in six languages for curating data for VLMs.

What's inside

Inside are 1.95 billion examples (166.7 TB) from 166 sources in English, Chinese, German, French, Japanese, and Korean in WebDataset tar archive format. This is an unfiltered candidate pool for the DataComp-VLM benchmark, and uploading part of the archives to the repository is still ongoing.

What you can do with it

  • Researching and testing multimodal data filtering algorithms before training VLMs
  • Large-scale pretraining of visual question answering (VQA) and captioning models
Tasks
Image text → text, Visual question answering
Languages
English, Chinese, German, French, Japanese, Korean, Latin, Finnish and 4 more
Size
1B–10B rows
Downloads this month
123.8K
Likes
3

How to load it

from datasets import load_dataset

ds = load_dataset("mlfoundations/dcvlm_pool_large")