Skip to content

Dataset · bigcode

Deduplicated source code corpus

bigcode/the-stack-dedup

Code in many programming languages with permissive licenses after near-deduplication, about 1.5 TB.

What's inside

The Stack is a source code corpus collected from repositories with permissive licenses. Version 1.0 had 30 programming languages and 18 licenses; after near-deduplication, the size reached 1.5 TB. Version 1.1 excluded weak copyleft MPL/EPL/LGPL and expanded the list to 193 licenses.

What you can do with it

  • Pre-training code LLMs
  • Fine-tuning code completion models
  • Checking license filtering
Tasks
Text generation
Languages
Code
Size
100M–1B rows
Downloads this month
14.1K
Likes
431
Access
on request on Hugging Face

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-dedup")