Dataset · mlfoundations
Huge raw data pool for VLM training (483M)
mlfoundations/dcvlm_pool_medium
Unfiltered 41.1 TB multimodal corpus in six languages for VLM data curation experiments.
What's inside
The dataset contains 483,576,747 examples totaling 41.1 TB from 166 sources in WebDataset tar archive formats in English, Chinese, German, French, Japanese, and Korean. This is a medium-sized raw, unfiltered pool from the DataComp-VLM benchmark, not intended for direct training without preliminary cleaning. Researchers set their own filters and mixing ratios to build a custom training set based on it.
What you can do with it
- Researching algorithms for multimodal data filtering, deduplication, and balancing
- Building custom training sets for image-text-to-text and VQA models
- Tasks
- Image text → text, Visual question answering
- Languages
- English, Chinese, German, French, Japanese, Korean, Latin, Finnish and 4 more
- Size
- 100M–1B rows
- Downloads this month
- 32.8K
- Likes
- 0
How to load it
from datasets import load_dataset
ds = load_dataset("mlfoundations/dcvlm_pool_medium")