Qwen3.5-4B-Kazakh-OptimizedTokenizer
base model with an optimized tokenizer for Kazakh and Russian
2 min readupdated
Developed by ISSAI, this model continues pretraining Qwen/Qwen3.5-4B-Base with 16,000 new tokens added for Kazakh and Russian. Expanding the tokenizer makes Kazakh text process nearly twice as fast and Russian more efficient (using 10% fewer tokens than the original), while English performance remains unchanged. As a base (pretrained) model, it completes text rather than following chat instructions, making it a solid foundation for further fine-tuning and building chatbots.
Key features
- Faster Kazakh processing: Kazakh text requires 1.98x fewer tokens compared to the base Qwen3.5.
- Russian language support: Added Russian tokens and pretraining data, improving metrics in Russian.
- Higher benchmark scores: Achieves the highest average scores on Kazakh tasks in the ISSAI lineup alongside strong Russian results.
- Qwen3.5 architecture: Hybrid architecture (24 Gated DeltaNet layers + 8 full-attention layers) with a hidden dimension of 2,560.
What you can do
- Chat fine-tuning: Use it as a foundation for building bilingual Russian-Kazakh assistants via fine-tuning.
- Text generation and completion: Apply it to autocomplete tasks and document processing in Kazakh and Russian.
- Pipeline integration: Embed the model into services with tight context limits to save input and output tokens.
Example
Text generation example using the transformers library:
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="cuda")
ids = tok("Қазақстан Республикасының астанасы", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))
Read more
DetailsArchitecture: 4.25B parameters, 32 layers, context window up to 32,768 tokens.
Training: 8.4B tokens (40% Kazakh, 20% Russian, 40% English), 15 × H100, DeepSpeed ZeRO-2.
Limitations: Base model without dialogue tuning or safety guardrails.
License: Apache-2.0.










Comments