Skip to content

Dataset · secemp9

arXiv Complete: full snapshot of arXiv papers

secemp9/arxiv-complete

Snapshot of 3.1 million arXiv papers with metadata, files, and PDFs.

What's inside

The corpus includes an arXiv metadata snapshot, version history, submission files, and rendered documents. Coverage is 3,148,796 papers; PDFs are available for 99.47% of papers, and a non-zero size is specified for 99.54% of versions. Metadata is sourced from OAI-PMH arXivRaw, and files are from a GCS mirror, S3 archives, and direct PDF downloads. This is a one-time snapshot with potential coverage gaps.

What you can do with it

  • Train retrieval models on scientific texts
  • Build a corpus for language modeling on arXiv
  • Test OCR/parsing of PDFs and metadata
Tasks
Text generation, Text retrieval
Languages
English
Size
100M–1B rows
Downloads this month
116.5K
Likes
804

How to load it

from datasets import load_dataset

ds = load_dataset("secemp9/arxiv-complete")