SWIFT Step Probe for Qwen3-8B
reasoning step verifier
2 min readupdated
This lightweight step verifier pairs a 32,770-parameter linear head with a frozen Qwen/Qwen3-8B base model. Built on the SWIFT methodology and trained on PRM800K datasets to evaluate external generations, it scores step-by-step math and logic solutions, calculating a reward for each step to pinpoint the very first mistake in a reasoning chain. It is a useful tool for researchers and developers working to improve the reliability and verifiability of large language models (LLMs).
Key features
- Lightweight linear head (just 32,770 parameters) built on top of a frozen Qwen3-8B.
- Evaluates each solution step by analyzing the last four hidden states (layers 33–36).
- Predicts the index of the first error in a reasoning chain, or returns -1 if the solution is error-free.
- Trained on filtered PRM800K data using step-level binary cross-entropy (BCE).
What you can do
- Score step-by-step math solutions generated by other LLMs on benchmarks like ProcessBench.
- Integrate the verifier into generation quality evaluation pipelines as a Process Reward Model (PRM).
- Run evaluation scripts via the SWIFT repository using local weights and configurations.
Example
git clone -b step-probe [https://github.com/aster2024/SWIFT.git](https://github.com/aster2024/SWIFT.git) && cd SWIFT
python step/eval_processbench.py Qwen/Qwen3-8B out.jsonRead more
DetailsBenchmarks: 28.9 average F1 on seed 42 (GSM8K: 36.8, MATH: 31.5, OlympiadBench: 23.9, Omni-MATH: 23.6).
Requirements: runs on top of Qwen/Qwen3-8B with a 0.45 activation threshold.
Training: 1 epoch, AdamW (lr 1e-4, weight decay 1e-5), up to 16,384 tokens per batch.










Comments