Prelude 9B SFT v2 Hallucination R2
an honest language model against hallucinations
2 min readupdated
A new research checkpoint (intermediate model version) with 9.09 billion parameters based on marianna13/prelude-9b-sft-v2 is trained to honestly admit when it does not know something instead of making up facts across ten European languages. The base model tends to invent biographies and plots for non-existent people or objects, whereas this version politely declines to answer and asks for context in 98% of cases. The model was created as part of OpenEuroLLM research and will be useful for developers who care about factual accuracy and minimizing fabrications in chat bots.
Key features
- Sharp reduction in fabrications: on fictitious topics, the hallucination rate dropped from 97.7% to 1.8%.
- Support for 10 European languages (English, German, French, Spanish, Italian, and others).
- Multi-stage training using SFT (supervised fine-tuning) and DPO (direct preference optimization).
- Retention of correct answers to known facts at 53.8% (though the model has become overly cautious).
What you can do
- Test the model's resistance to fabrications (hallucinations) using your own tricky questions in different languages.
- Integrate the model into text generation pipelines where refusing to answer in the absence of reliable data is critically important.
- Use the standard Prelude ChatML chat template and tokenizer from the Qwen3 architecture for quick integration.
- Explore the datasets and pipeline on GitHub to replicate experiments on combating hallucinations.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "birgermoell/prelude-9b-sft-v2-hallucination-r2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "Wer ist Declan Monmonley?"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False,
return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))Read more
DetailsBenchmarks (500 questions): hallucinations on fictitious subjects decreased from 97.7% to 1.8%, and the truthfulness score increased from -85.2 to +3.8.
Requirements: two nodes with 4 × A100 (64 GB) for training, and inference requires a modern video card with bfloat16 support.
Limitations: the model has become too cautious and now frequently refuses to answer real questions about details of famous people (such as birth dates), and the thinking mode has lost factual accuracy.










Comments