Skip to content

Dataset · bigcode

Cleaned code identifiers from The Stack v2

bigcode/the-stack-v2-train-smol-ids

Subsample of deduplicated source code identifiers for training generative models.

What's inside

Contains tabular and text data in programming languages (10 to 100 million records). This is a compact version based on the deduplicated The Stack v2 corpus from the BigCode project, filtered by heuristics and grouped by repository.

What you can do with it

  • Pre-train models for code generation and auto-completion
  • Collect and filter a custom source code corpus by repository IDs
Tasks
Text generation
Languages
Code
Size
10M–100M rows
Downloads this month
830
Likes
65
Access
on request on Hugging Face

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-v2-train-smol-ids")