Skip to content

Dataset · tokyotech-llm

SwallowCode: EN/JA code corpus

tokyotech-llm/swallow-code

Large English and Japanese code corpus for training and fine-tuning language models.

What's inside

Dataset for English and Japanese text generation, volume ranges from 100M to 1B records; tabular data and text are specified in metadata. The card contains a warning: the ablation/exp1-the-stack-v2-train-smol-ids-python subset was removed due to potentially unsafe data from the Python portion of The Stack v2. ClamAV also flagged Win.Trojan.MSShellcode-88; the set can be restored from the-stack-v2-train-smol-ids.

What you can do with it

  • fine-tuning LLMs for code generation
  • code SFT in English and Japanese
  • checking data safety during code training
Tasks
Text generation
Languages
English, Japanese
Size
100M–1B rows
Downloads this month
5.2K
Likes
84

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("tokyotech-llm/swallow-code")