switchyard-decider
a fast prompt router for LLMs
2 min readupdated
switchyard-decider is a 0.4B-parameter fine-tune of answerdotai/ModernBERT-large built to score the complexity of prompts sent to coding assistants. It acts as a local router, gauging task difficulty so simple queries go to a cheaper, faster mid-tier model while tough ones get routed to a frontier model. Averaging 25 ms per response and hitting an AUC of 0.895 on English coding prompts, it replaces bulky LLM-based deciders.
Key features
- Fast inference: response latency sits around 23–25 ms (p50 on Apple Silicon MPS).
- Strong metrics: reaches an AUC of 0.895 on the validation dataset, beating several larger teacher models.
- Lightweight footprint: weights come in at roughly 792 MB (fp16), with memory usage around 560 MB.
- Domain-tuned: trained on ~153k prompts drawn from popular coding evaluation datasets.
What you can do
- Plug into an LLM gateway: automatically route incoming requests based on task complexity.
- Spin up an HTTP server: run a local prompt-scoring service using the built-in Python script.
- Cut API costs: lower spending on expensive frontier models by offloading straightforward tasks to cheaper backends.
Example
import numpy as np
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
d = "switchyard-decider"
tok = AutoTokenizer.from_pretrained(d)
model = AutoModelForSequenceClassification.from_pretrained(d, dtype=torch.float32).eval()
text = "Rename the variable foo to bar in utils.py"
with torch.no_grad():
lg = model(**tok(text, truncation=True, max_length=768, return_tensors="pt")).logits[0].float()
print(lg.shape)
Read more
DetailsBenchmarks: AUC 0.895 (v1.1 with a ridge probe), up to 0.904 on a fixed OpenRouter subset.
Hardware requirements: ~560 MB VRAM/RAM; runs on CPU, Apple Silicon, or GPU.
Limitations: English only, focused strictly on coding assistants and chat prompts.
Training details: lr 3e-5, max length 768, bf16, trained on an NVIDIA RTX 5070 Ti.










Comments