DeepSeek-V4.1-Flash-1017-singprobe
Built-in Streaming Guardrail for Language Model
2 min readupdated
SingProbe is an intrinsic streaming guardrail for the DeepSeek-V4.1-Flash-1017 model. The module reads hidden states of the base model at specific generation layers and evaluates user intent, response safety, and hallucination risk in real-time for every token. The weights are trained in bfloat16 format and contain 10.10M parameters. The release will be useful for developers who need reliable filtering of dangerous content and on-the-fly hallucination detection during inference.
Key features
- Reading three internal layers of the base model (12, 25, 39) directly during token generation.
- Evaluation of 8 different user intent variants, plus separate metrics for response safety and hallucinations.
- Use of 8 attention heads with a dimension of 64 and a window of 2048 tokens.
- Running via a special SGLang branch for efficient parallel inference.
What you can do
- Deploy a secured generation server based on SGLang to filter toxic or dangerous content in real-time.
- Track the hallucination risk level directly in the
meta_info.token_probe_probsfield of API responses. - Check incoming user request intents across 8 categories for flexible dialog routing.
Example
Running the server with the connected probe via SGLang:
python3 -m sglang.launch_server \
--model-path /home/admin/DeepSeek-V4.1-Flash-1017 \
--probe-ckpt /yijian3/536716/SingGuardStream-Final/release/DeepSeek-V4.1-Flash-1017-singprobe \
--trust-remote-code --tp 8 --ep-size 8 \
--context-length 16384 --port 30000Read more
Details• Probe parameters: 10.10M, layers [12, 25, 39] utilized
• Request intent classification (F1): 0.8756
• Response safety (F1): 0.8668
• Streaming safety (Response AUC / token AUC): 0.9907 / 0.9522
• Hallucination detection (AUC): 0.7951
• Requirements: 8 × H20 GPU, special SGLang branch 261029










Comments