Skip to content

Dataset · bigcode

The Stack source code corpus

bigcode/the-stack

A large source code corpus in dozens of programming languages with permissive licenses.

What's inside

This is source code from GitHub: v1.0 featured 30 programming languages and 18 permissive licenses, totaling 3 TB after near-deduplication. In v1.1, weak copyleft licenses MPL/EPL/LGPL are excluded, and the list of permissive licenses is expanded to 193. The data is presented as tabular and text, with code as the language.

What you can do with it

  • Train LLMs for code auto-completion
  • Research licenses and code filtering
  • Fine-tune a code model on permissive code
Tasks
Text generation
Languages
Code
Size
100M–1B rows
Downloads this month
115.1K
Likes
1.1K
Access
on request on Hugging Face

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("bigcode/the-stack")