tiny-aya-global-attribution-lora
stopping LLMs from guessing who did it
2 min readupdated
A research LoRA adapter (a lightweight add-on for a base model) by vincentnkq for the multilingual language model CohereLabs/tiny-aya-global. When asked who performed an action that isn't explicitly stated in a short passage, the base model often guesses one of the mentioned characters at random. This adapter trains the model to honestly admit when the text lacks the necessary information while still answering accurately whenever the facts are actually there. Ideal for safety researchers and developers testing how resilient compact LLMs are to biased assumptions during text analysis.
Key features
- No more guessing: the rate at which the model randomly names a person in ambiguous contexts dropped from 59.7% to 1.3% on average across four languages (English, Spanish, Dutch, and Turkish).
- Preserved accuracy: on paired questions with unambiguous answers, accuracy rose from 69.2% to 80.7%, while twin accuracy jumped from 24.7% to 79.5%.
- Cross-lingual transfer: training ran exclusively on an English dataset (1,412 examples), yet the effect transferred to other languages, including Russian, French, Hindi, and Arabic.
- Resilience without answer choices: even in an open-ended format without A/B/C multiple-choice hints, the model names a person on ambiguous questions in only 8.1% of cases, down from 61.5% for the base model.
What you can do
- Reproduce debiasing experiments: evaluate how
tiny-aya-globalbehaves on BBQ and MBBQ benchmarks (datasets designed to measure social stereotypes in question-answering systems). - Use it as an SFT tuning reference: take this LoRA configuration (rank 16, 1,412 rows) as a starting point to train small models to abstain when context is missing.
Example
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
base = "CohereLabs/tiny-aya-global"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "vincentnkq/tiny-aya-global-attribution-lora")
Read more
DetailsBenchmarks: twin accuracy on the English BBQ test jumped from 34.7% to 92.7% (+58.0 percentage points); on the machine-translated Russian version, it reached 86.9% with only 4.8% false attributions.
Training: LoRA rank 16, alpha 32, dropout 0.05 (applied to q, k, v, o, gate, up, and down_proj modules), 3 epochs (267 steps), lr 0.0002, bfloat16; took 37 minutes on a single L4 GPU ($0.49).
Software and limitations: tested on transformers 5.22.1 and peft 0.21.2; this is a research artifact, not a general-purpose safety filter or universal hallucination guard for other task types.










Comments