Dataset · codeparrot
115M GitHub code files
codeparrot/github-code-clean
A filtered version of GitHub code containing 115M files across 32 programming languages, totaling nearly 1 TB of text.
What's inside
This is a more filtered version of the codeparrot/github-code dataset. It includes 115M GitHub code files across 32 programming languages and 60 extensions, totaling nearly 1 TB of textual data.
What you can do with it
- Pre-training code LLMs
- Training code auto-completion
- Researching GitHub code filtering
- Size
- 10M–100M rows
- Downloads this month
- 53.4K
- Likes
- 151
Models trained on it
- ibm-granite/granite-3b-code-base-2kText generation 40.5K
- mradermacher/granite-8b-code-base-128k-i1-GGUF 3.5K
- mradermacher/granite-3b-code-base-i1-GGUF 3.2K
- ibm-granite/granite-8b-code-base-4kText generation 2.9K
- mradermacher/granite-34b-code-base-8k-i1-GGUF 2.7K
- mradermacher/granite-3b-code-base-2k-i1-GGUF 2.5K
- codeparrot/codeparrot-small-multiText generation 1.4K
- ibm-granite/granite-20b-code-base-8kText generation 1.4K
How to load it
from datasets import load_dataset
ds = load_dataset("codeparrot/github-code-clean")