OpenCode BPE-latent LoRA checkpoints
latent reasoning adapter set for Qwen3-4B
2 min readupdated
A set of LoRA adapters (small add-ons for the base model) for the Qwen3-4B language model, trained on a chain-of-thought (CoT) re-encoded into a lossless BPE latent vocabulary. Chains of thought are compressed using special latent tokens, enabling experiments with code generation efficiency. The checkpoints contain embedding weights, tokenizers, and configuration dictionaries for various token merging rules. The set targets developers and researchers studying hidden LLM reasoning in code generation.
Key features
- Base on Qwen3-4B: adapters built on top of the SFT model with natural CoT.
- Latent encoding: use of BPE vocabulary losses under various never-merge rules.
- Multiple variants: folders contain different compression ratios (1.2x, 1.5x, etc.) and data filtering options (comments / no-comment).
- LiveCodeBench evaluation: code generation benchmarks performed for two-stage decoding.
What you can do
- Apply LoRA adapters from the repository on top of base Qwen3-4B for code generation experiments.
- Explore the impact of various never-merge rules (such as blocking numbers or keywords) on LCB metrics.
- Analyze the difference between code comment datasets (cot) and non-comment datasets (cot_nc).
Example
Model passing LiveCodeBench (LCB) evaluation with two-stage decoding (latent cap up to 6192, code cap up to 2000, temperature 0.2) to verify code generation accuracy.
Read more
DetailsBase model: Qwen3-4B
Fine-tuning parameters: 3 epochs, max-len 8192, LoRA r16/a32, lr 2e-4, seed 42
Best LCB v5 results reach 55.5 (nc_ident) and 32.6 (nc_none, nc_tok) in v6new










Comments