Dataset · tokyotech-llm
SwallowCode: EN/JA code corpus
tokyotech-llm/swallow-code
Large English and Japanese code corpus for training and fine-tuning language models.
What's inside
Dataset for English and Japanese text generation, volume ranges from 100M to 1B records; tabular data and text are specified in metadata. The card contains a warning: the ablation/exp1-the-stack-v2-train-smol-ids-python subset was removed due to potentially unsafe data from the Python portion of The Stack v2. ClamAV also flagged Win.Trojan.MSShellcode-88; the set can be restored from the-stack-v2-train-smol-ids.
What you can do with it
- fine-tuning LLMs for code generation
- code SFT in English and Japanese
- checking data safety during code training
- Tasks
- Text generation
- Languages
- English, Japanese
- Size
- 100M–1B rows
- Downloads this month
- 5.2K
- Likes
- 84
Models trained on it
- tokyotech-llm/Llama-3.1-Swallow-8B-v0.5 2.1K
- mradermacher/Llama-3.1-Swallow-8B-v0.5-i1-GGUF 738
- mradermacher/Llama-3.1-Swallow-8B-v0.5-GGUF 179
- tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500 71
- tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500 54
- tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000 39
- tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0002500 36
- tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000 33
How to load it
from datasets import load_dataset
ds = load_dataset("tokyotech-llm/swallow-code")