Dataset · bigcode
Deduplicated source code corpus
bigcode/the-stack-dedup
Code in many programming languages with permissive licenses after near-deduplication, about 1.5 TB.
What's inside
The Stack is a source code corpus collected from repositories with permissive licenses. Version 1.0 had 30 programming languages and 18 licenses; after near-deduplication, the size reached 1.5 TB. Version 1.1 excluded weak copyleft MPL/EPL/LGPL and expanded the list to 193 licenses.
What you can do with it
- Pre-training code LLMs
- Fine-tuning code completion models
- Checking license filtering
- Tasks
- Text generation
- Languages
- Code
- Size
- 100M–1B rows
- Downloads this month
- 14.1K
- Likes
- 431
- Access
- on request on Hugging Face
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("bigcode/the-stack-dedup")