Dataset · secemp9
arXiv Complete: full snapshot of arXiv papers
secemp9/arxiv-complete
Snapshot of 3.1 million arXiv papers with metadata, files, and PDFs.
What's inside
The corpus includes an arXiv metadata snapshot, version history, submission files, and rendered documents. Coverage is 3,148,796 papers; PDFs are available for 99.47% of papers, and a non-zero size is specified for 99.54% of versions. Metadata is sourced from OAI-PMH arXivRaw, and files are from a GCS mirror, S3 archives, and direct PDF downloads. This is a one-time snapshot with potential coverage gaps.
What you can do with it
- Train retrieval models on scientific texts
- Build a corpus for language modeling on arXiv
- Test OCR/parsing of PDFs and metadata
- Tasks
- Text generation, Text retrieval
- Languages
- English
- Size
- 100M–1B rows
- Downloads this month
- 116.5K
- Likes
- 804
How to load it
from datasets import load_dataset
ds = load_dataset("secemp9/arxiv-complete")