Spark 2.0-code
A compact model for Python code generation
2 min readupdated
Spark 2.0-code is a small language model with 322 million parameters, trained entirely from scratch for writing Python code. Built by an independent developer on a single RTX 3090 graphics card, it uses a hybrid architecture featuring Gated DeltaNet (fast linear attention layers). The model takes a task in English and generates ready-to-use code. It handles individual functions and short console scripts well, and also supports a chat mode for fixes based on error messages.
Key features
- Architecture: A hybrid design combining Gated DeltaNet layers with full attention every fourth layer, supporting a context of up to 4096 tokens.
- Training: Pre-trained on 18.4 billion Python tokens using Muon and AdamW optimizers, followed by two stages of SFT.
- Custom tokenizer: A specialized BPE tokenizer with a vocabulary size of 16,384 tokens, optimized for Python.
- Workflow: Generation using greedy decoding (temperature 0) via structured prompts with user and assistant formatting.
What you can do
- Generate functions: Quickly create small Python scripts, algorithms, and individual functions from text descriptions.
- Write console utilities: Build simple interactive programs featuring menus, loops, and basic logic.
- Fix errors: Send the model code along with an error message for automated refinement and debugging.
Example
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py "Write a Python function that checks whether a number is prime."Read more
DetailsBenchmarks: HumanEval pass@1 is 47.6% (the first stage hit 52.4%, but the second stage handles complete programs better)
Requirements: An NVIDIA graphics card with CUDA support (requires the flash-linear-attention library)
Limitations: Struggles more with complex projects of 100–200 lines and geometry; knows Python only
License: CC BY-NC 4.0 (non-commercial, due to the use of the KodCode dataset)










Comments