Skip to content

Dataset · mlfoundations

Chunk of the massive MINT-1T multimodal corpus (PDF)

mlfoundations/MINT-1T-PDF-CC-2023-40

Part of the open MINT-1T dataset with interleaved text and images from English PDFs and research papers.

What's inside

A fragment of the large-scale open MINT-1T corpus created by University of Washington researchers for multimodal pre-training. The overall project combines 1 trillion text tokens and 3.4 billion images, including data from previously unused PDF documents and ArXiv articles; this repository contains 100 billion to 1 trillion English records. File format is not specified in the description.

What you can do with it

  • Pre-training large multimodal models (VLMs) on text-image pairs
  • Training systems to recognize and understand complex documents, PDFs, and research papers
Tasks
Image → text, Text generation
Languages
English
Size
100B–1T rows
Downloads this month
29.6K
Likes
9

How to load it

from datasets import load_dataset

ds = load_dataset("mlfoundations/MINT-1T-PDF-CC-2023-40")