LLaDA2.2-mini-Thinking: A reasoning model for block-diffusion MoE
2 min read
This model is an upgraded version of LLaDA2.2-mini (a 16B block-diffusion MoE ( Mixture of Experts ) with 1.4B active parameters), fine-tuned to reason inside … tags before generating a response or calling tools. The base model answers immediately without reasoning in text. The new version is trained on agent sessions and reasoning examples from DeepSeek, combining generation capabilities with logical task analysis.
Key features
- Added reasoning before the response: the model explicitly writes out its thought process in
tags. - Retained native support for tool calls from the base architecture.
- Trained on 2,003 agent task sessions and 3,000 reasoning examples (25.6M tokens total).
- Weights merged in bf16 format, training took 5.3 hours on a single A100 80GB GPU.
What you can do
- Run a local server via SGLang with a deepseek-r1 style reasoning parser.
- Integrate the model into assistant and agent tasks that require step-by-step task breakdown before action.
- Use standard Transformers code to generate responses with a preliminary thought block.
- Test the model on math and coding tasks requiring detailed logical inference.
Example
Running response generation with logical inference in Python using transformers:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "kaivoss/LLaDA2.2-mini-Thinking"
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")
messages = [{"role": "user", "content": "Solve 5x^2 - 6x + 4 = 0."}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to("cuda")
out = model.generate(ids, gen_length=1536, block_length=32, steps=32, threshold=0.5, eos_early_stop=True)
text = "\n" + tok.decode(out[0], skip_special_tokens=True)
thinking, answer = text.split("", 1)
print(answer.strip())Read more
DetailsBase: inclusionAI/LLaDA2.2-mini
Method: LoRA r32 on attention, dense MLPs, and shared experts with subsequent weight merging
VRAM requirements: about 33 GB for weights, an 80 GB VRAM GPU is sufficient
Limitations: evaluated on only 8 examples, long reasoning sequences may hit token length limits
Base model license: Apache-2.0










Comments