Dataset · bigcode
Deduplicated code corpus The Stack v2
bigcode/the-stack-v2-dedup
Nearly deduplicated version of The Stack v2—a large code corpus for LLM training, grouped by repositories.
What's inside
This is a version of bigcode/the-stack-v2 with additional near-deduplication. Data is grouped by repositories. The Stack v2 family also includes the full version, this deduplicated version, filtered train-full-ids with 600+ programming languages, and the reduced train-smol. The data language is code.
What you can do with it
- Pretraining code LLMs
- Fine-tuning models on cleaner code
- Evaluating deduplication quality for code corpora
- Tasks
- Text generation
- Languages
- Code
- Size
- 1B–10B rows
- Downloads this month
- 2.8K
- Likes
- 148
- Access
- on request on Hugging Face
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("bigcode/the-stack-v2-dedup")