SmallCoder (303M)
An ultracompact language model for code generation
2 min read
SmallCoder is a compact model with 303 million parameters, trained from scratch specifically for code generation and algorithmic reasoning. Despite its modest size, it achieves performance on par with much larger language models (LLMs), comparing favorably to 1B–7B parameter counterparts. The model is built on the LLaMA architecture and uses the bigcode/starcoder tokenizer. It is a great fit for developers looking for a fast, local model for writing and debugging code on limited hardware.
Key features
- High efficiency: delivers HumanEval results (27.4%) comparable to models several times its size.
- Trained from scratch: underwent a complete training course on 29.8 billion tokens, covering linguistic foundation, code, and math stages.
- Code specialization: optimized for Python and algorithmic tasks.
- Low system requirements: thanks to its 303M parameter size, it runs fast even on weak GPUs or CPUs.
What you can do
- Code generation: integrate into local development environments or bots to assist with writing Python functions.
- Autocomplete: use for quickly creating boilerplate code and algorithmic scripts.
- Learning and experiments: run and test language models on regular consumer PCs.
- Research: study approaches to efficient training of compact LLMs built from scratch.
Example
An example of function generation using the model's dialogue format:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "ITLL/smallcoder-303m"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(device)
prompt = """User: Write a Python function to compute Fibonacci numbers.
Assistant:"""
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))Read more
Details• Benchmarks: HumanEval (pass@1) — 27.4%, MBPP (pass@1) — 31.0%.
• Architecture: LLaMA-type causal decoder (24 layers, hidden size 768).
• Limitations: context up to 1024 tokens, struggles with general text and math outside of code.
• Training: 6B tokens during the SFT stage on TPU v4-32.










Comments