ettin-17m-thinking-ruler
evaluating prompt complexity for LLMs
2 min readupdated
A lightweight 17-million-parameter encoder model by agentlans, built on top of jhu-clsp/ettin-encoder-17m. The model analyzes a text prompt and outputs a continuous reasoning-length score that indicates how long a Chain-of-Thought (CoT) the language model will need to respond. The higher the score, the more complex the task. This tool is useful for agent and pipeline developers looking to dynamically allocate computing resources based on request complexity.
Key features
- Ultra-compact size (only 17M parameters) for fast execution on any hardware.
- Outputs a continuous complexity score for each incoming prompt.
- Trained on the
agentlans/think-at-lengthdataset using chains of thought from advanced models. - Fully compatible with the
transformerslibrary and PyTorch.
What you can do
- Dynamically determine user request complexity before sending it to a heavy LLM.
- Set up conditional routing: direct simple queries to fast models and complex ones to deep-reasoning models.
- Analyze incoming prompt traffic by complexity level.
Example
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "agentlans/ettin-17m-thinking-ruler"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()
prompts = [
"What is the capital of Canada?",
"Prove that there are infinitely many primes and discuss edge cases in formal proofs.",
]
inputs = tokenizer(prompts, padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
scores = model(**inputs).logits.squeeze(-1)
for prompt, score in zip(prompts, scores.tolist()):
print(f"Score: {score:.4f} | Prompt: {prompt}")
Read more
DetailsBenchmarks: best validation MSE score (0.3825) achieved at epoch 2 of training.
Limitations: the result is a relative complexity signal rather than a direct token count; final reasoning length heavily depends on the target model's generation settings.
Training details: Learning rate 5e-5, batch size 8, 3 epochs, AdamW optimizer, bfloat16, NEFTunes alpha = 5.










Comments