Dataset · open-index
Complete metadata of nearly 3 million arXiv papers
open-index/open-arxiv
Metadata of 2.99 million scientific publications from 1991 to 2026 in parquet format for search and analytics.
What's inside
Contains metadata for 2.99 million arXiv research papers over 40 years (1991 to 2026): titles, abstracts, authors, categories, DOIs, and version history. The data is sourced from Cornell University's Kaggle dataset and converted into 417 Zstd-compressed parquet shards, primarily in English with multilingual inclusions.
What you can do with it
- Training scientific text classifiers and abstract generators
- Building semantic search and feature extraction systems
- Analyzing science trends and creating paper-based question-answering systems
- Tasks
- Text classification, Text generation, Embeddings, Question answering
- Languages
- English, Multilingual
- Size
- 1M–10M rows
- Downloads this month
- 2.9K
- Likes
- 21
How to load it
from datasets import load_dataset
ds = load_dataset("open-index/open-arxiv")