Dataset · mlfoundations
DataComp-VLM multilingual image-text pool
mlfoundations/dcvlm_pool_large
Huge raw pool of nearly 2 billion image-text pairs in six languages for curating data for VLMs.
What's inside
Inside are 1.95 billion examples (166.7 TB) from 166 sources in English, Chinese, German, French, Japanese, and Korean in WebDataset tar archive format. This is an unfiltered candidate pool for the DataComp-VLM benchmark, and uploading part of the archives to the repository is still ongoing.
What you can do with it
- Researching and testing multimodal data filtering algorithms before training VLMs
- Large-scale pretraining of visual question answering (VQA) and captioning models
- Tasks
- Image text → text, Visual question answering
- Languages
- English, Chinese, German, French, Japanese, Korean, Latin, Finnish and 4 more
- Size
- 1B–10B rows
- Downloads this month
- 123.8K
- Likes
- 3
How to load it
from datasets import load_dataset
ds = load_dataset("mlfoundations/dcvlm_pool_large")