dwiiittt/telaah-tor-qwen3-4b-instruct-2507
Project documentation AI auditor
2 min readupdated
The model evaluates the logic and completeness of TOR/KAK sections (terms of reference and concepts) based on the PMK 41/2026 checklist of criteria. The output provides a structured JSON object with a verdict for each item, reasoning with quotes from the text, and specific recommendations for improvement. The project is built on the compact Qwen/Qwen3-4B-Instruct-2507 architecture using LoRA (a small trainable add-on to the base model) and trained on a specialized dataset of ministerial documents. The tool will be useful for document reviewers, analysts, and automated government audit system developers.
Key features
- Automated verification of the logic of key sections (justification, strategy, risks, costs).
- High evaluation accuracy on the test dataset with a balanced accuracy of over 95%.
- Strict JSON output format for results without extra text or syntax errors.
- Excellent balance of compact size (4 billion parameters) and performance on par with larger models.
What you can do
- Integrate the model into an enterprise pipeline for reviewing project documents and terms of reference.
- Deploy an automated bot for the initial review of submissions and reports.
- Run locally via vLLM for batch processing of dozens of text files.
- Use as an assistant to search for logical inconsistencies in project plans.
Example
Example of using the model to review terms of reference via the transformers library:
import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "dwiiittt/telaah-tor-qwen3-4b-instruct-2507"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16, device_map="auto").eval()
messages = [{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_TEXT}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=768, do_sample=False)
print(json.loads(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True)))Read more
Details• Metrics: F1 for the "illogical" class is 0.969, overall document accuracy is 96.5%.
• Hardware: trained on 1× NVIDIA A100 (80 GB), requires bf16 support and transformers>=4.51 libraries.
• Limitations: struggles more with checking remote RO volumes and general descriptions without dedicated headings; it is an auxiliary tool rather than a human replacement.
• Architecture: SFT with LoRA (all-linear, r=16, alpha=32) based on Qwen3-4B-Instruct-2507.










Comments