Dataset · open-index
Complete Reddit post and comment archive from 2005 to 2026
open-index/arctic
Massive English corpus of 15.7 billion Reddit records in parquet format for text training and analysis.
What's inside
The Arctic Shift project archive includes all public subreddits from December 2005 to February 2026 in English. It contains 15.7 billion records (12.9 billion comments and 2.8 billion posts) packed into 1.3 TB of monthly-partitioned compressed parquet files.
What you can do with it
- Large-scale pretraining of English language models on natural conversational speech
- Training text classification and feature extraction models
- Studying the evolution of topics and public discussions over twenty years
- Tasks
- Text generation, Text classification, Embeddings
- Languages
- English
- Size
- 1B–10B rows
- Downloads this month
- 42.8K
- Likes
- 42
How to load it
from datasets import load_dataset
ds = load_dataset("open-index/arctic")