Signpost-Jev-4B
a model for structured decisions and benchmark evaluation
2 min readupdated
Signpost-Jev-4B is a specialized model built on Qwen3.5-4B, trained to make structured decisions following the JevBench standard. It takes a state (as plain text or a JSON record) along with typed questions, and outputs probabilities for each answer option (true/false, multiple choice, or scales). Instead of generating free-form text, the model computes each decision in a single pass through the answer tokens. It is optimized to run with vLLM and is built for developers working on automated decision-making and data evaluation systems.
Key features
- Purpose-built: fine-tuned for JevBench-style decision-making without generating unnecessary text.
- Low latency: delivers results with a median latency of 44 ms per request on an L40S GPU.
- Architecture: based on Qwen3.5-4B using LoRA (a lightweight adapter on top of the base model) at rank 16 in fp16, with the weights merged afterward.
- Accuracy: achieves high accuracy on public JevBench datasets (up to 0.979 on easy tasks).
What you can do
- Deploy a service: self-host the model via vLLM to handle decision-making requests.
- Analyze structured data: feed in JSON records or text to score them against defined criteria and scales.
- Test hypotheses: use it as an automated judge to verify structured outputs.
Example
Running the model via a vLLM server:
vllm serve skamalj/signpost-jev-4b --max-model-len 32768 --max-num-seqs 8 --logprobs-mode raw_logprobsRead more
DetailsJevBench benchmark accuracy: easy — 0.979 (48 items), original — 0.875 (72 items), hard — 0.730 (111 items), overall — 0.827.
Median latency: 44 ms per decision (vLLM on a single L40S).
Training: 2 epochs on the full data mix, followed by 1 short epoch on hard records.
License: Apache-2.0.










Comments