Skip to content

Dataset · open-index

Complete metadata of nearly 3 million arXiv papers

open-index/open-arxiv

Metadata of 2.99 million scientific publications from 1991 to 2026 in parquet format for search and analytics.

What's inside

Contains metadata for 2.99 million arXiv research papers over 40 years (1991 to 2026): titles, abstracts, authors, categories, DOIs, and version history. The data is sourced from Cornell University's Kaggle dataset and converted into 417 Zstd-compressed parquet shards, primarily in English with multilingual inclusions.

What you can do with it

  • Training scientific text classifiers and abstract generators
  • Building semantic search and feature extraction systems
  • Analyzing science trends and creating paper-based question-answering systems
Tasks
Text classification, Text generation, Embeddings, Question answering
Languages
English, Multilingual
Size
1M–10M rows
Downloads this month
2.9K
Likes
21

How to load it

from datasets import load_dataset

ds = load_dataset("open-index/open-arxiv")