Skip to content

Dataset · permutans

Metadata for 2.5M arXiv papers by subject and date

permutans/arxiv-papers-by-subject

A reorganized corpus of English arXiv scientific paper metadata broken down by subject, year, and month for selective downloading.

What's inside

The dataset collects text metadata for over 2.5 million English scientific publications from the arXiv portal. It is a reworked version of another repository where files are organized in a folder hierarchy by discipline codes, years, and months, enabling users to download only the necessary slices without downloading the entire archive.

What you can do with it

  • Train text generation and feature extraction models in narrow scientific domains
  • Gather thematic publication samples for specific periods for analytics and RAG
Tasks
Text generation, Embeddings
Languages
English
Size
1M–10M rows
Downloads this month
102.6K
Likes
39

How to load it

from datasets import load_dataset

ds = load_dataset("permutans/arxiv-papers-by-subject")