Skip to content

Dataset · open-index

Complete Reddit post and comment archive from 2005 to 2026

open-index/arctic

Massive English corpus of 15.7 billion Reddit records in parquet format for text training and analysis.

What's inside

The Arctic Shift project archive includes all public subreddits from December 2005 to February 2026 in English. It contains 15.7 billion records (12.9 billion comments and 2.8 billion posts) packed into 1.3 TB of monthly-partitioned compressed parquet files.

What you can do with it

  • Large-scale pretraining of English language models on natural conversational speech
  • Training text classification and feature extraction models
  • Studying the evolution of topics and public discussions over twenty years
Tasks
Text generation, Text classification, Embeddings
Languages
English
Size
1B–10B rows
Downloads this month
42.8K
Likes
42

How to load it

from datasets import load_dataset

ds = load_dataset("open-index/arctic")