Skip to content

Dataset · codeparrot

Cleaned Python Code from GitHub

codeparrot/codeparrot-clean

Deduplicated GitHub Python file corpus for training code generation models.

What's inside

This is a cleaned version of the CodeParrot dataset: Python files from GitHub. Exact duplicates and noise were removed from the source data by limiting line length and alphanumeric character ratio, and removing autogenerated files. Modalities include tabular data and text; size ranges from 1 to 10 million records.

What you can do with it

  • Pretraining small code LLMs
  • Fine-tuning Python auto-completion models
  • Evaluating code cleaning quality
Size
1M–10M rows
Downloads this month
33.1K
Likes
92

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("codeparrot/codeparrot-clean")