RootML/TinyStories-33M-Wsp-Arithmetic-Claude-Fable-5-2
patching model weights for arithmetic
2 min readupdated
Built on roneneldan/TinyStories-33M (a small 33M-parameter language model trained on children's stories), this release uses the experimental WSPMathBench (weight-space programming) method. Instead of traditional fine-tuning, another frontier LLM wrote a single Python script to directly overwrite part of the weights, hardcoding a hand-crafted arithmetic circuit. The result is a research artifact that solves basic addition and multiplication problems, though it is no longer a general-purpose language model. It's a solid fit for researchers studying neural network weights and model architectures.
Key features
- Single-step arithmetic circuit injection with zero training or backpropagation.
- Modifies 2.39 million weights (about 8% of the total) across the model's first two layers.
- Boosts accuracy on a test set of simple arithmetic problems from 4.5% to 90.4%.
What you can do
- Run inference with the transformers library to test the injected circuit's outputs.
- Use the weights for weight-space programming research and analyzing hidden circuits inside LLMs.
Example
import re
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "RootML/TinyStories-33M-WSP-arithmetic-claude-fable-5-2"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16).eval()
question = "38 + 47 ="
ids = tokenizer(question, return_tensors="pt", add_special_tokens=False).input_ids
with torch.no_grad():
out = model.generate(ids, attention_mask=torch.ones_like(ids), max_new_tokens=16,
do_sample=False, pad_token_id=tokenizer.eos_token_id)
text = tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
print(re.search(r"-?\d+", text).group())
Read more
DetailsAccuracy across all 500 questions: 90.4% (up from 4.5% on the base model).
Symbolic math prompts (`7*5=`): 100% exact match.
Word-based math prompts (`seven times five equals`): 80.8% exact match.
Limitations: not a general-purpose language model; outputs random noise right after answering the math problem.
Weight format: safetensors (bfloat16).










Comments