cybersos-tiny-aya-grpo-lora
an RL adapter for step-by-step account recovery
2 min readupdated
This is a LoRA (a lightweight add-on on top of a base model) for CohereLabs/tiny-aya-global by pgdev0410, trained to guide users through recovering access to personal Google accounts and Android devices. At each step, the model selects a single action: clarify an access factor, simulate a supported recovery step, or wrap up the scenario with a confirmed outcome. Both training and evaluation ran entirely inside a deterministic simulator—the model never connects to real accounts. Unlike the base model and the earlier SFT release, this version was fine-tuned with GRPO (group-relative policy optimization) on tricky edge cases where SFT stumbled, improving accuracy and cutting down on unsafe suggestions. It's a solid fit for developers and researchers exploring agentic pipelines and safe decision-making in tech support.
Key features
- Step-by-step agent logic: the model evaluates the current state and picks strictly one action per turn.
- SFT + GRPO fine-tuning: initialized from the
cybersos-tiny-aya-sft-loracheckpoint and further trained on hard cases using AgentTune GRPO. - Improved safety: episodes containing critically unsafe advice dropped from 22 to 13, and a safety guard blocked every single one of those attempts.
- Reproducibility: prompt contracts, tool schemas, and generation parameters are locked down in the project repository.
What you can do
- Run it in research sandboxes and tech support simulators to benchmark multi-step AI agents.
- Compare how the SFT and GRPO checkpoints behave across the exact same account recovery scenarios.
- Integrate it into an English-language account recovery assistant prototype—paired strictly with an external safety guard.
Example
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global", revision="af89d219b53ed9b13b8a4645f9c8028973510324", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "pgdev0410/cybersos-tiny-aya-grpo-lora")
tokenizer = AutoTokenizer.from_pretrained("pgdev0410/cybersos-tiny-aya-grpo-lora")
Read more
DetailsBenchmark (240 episodes): 223 of 240 goals completed (compared to 211 for both the SFT version and baseline planner, and 0 for the raw base model).
Safe completion of recoverable cases: 58/68; proper handling of out-of-scope boundaries: 165/172.
Training: 20 GRPO steps (8 rollouts per scenario, temperature 1.0, lr 1e-5, KL β 0.04, bf16) on a single H200 GPU.
Limitations: English only, 16 verified rules, not intended for use with live accounts, non-commercial CC-BY-NC-4.0 license.










Comments