da-emb-anneal: A Norwegian language model with annealing
2 min read
The da-emb-anneal model is a 2.12B parameter base LLM (large language model) built as a student project at the University of Oslo using the Qwen3 architecture. It was derived from a Danish base model by replacing the tokenizer with a Norwegian one (32k) and performing additional training (annealing) on high-quality texts in Norwegian Bokmål and Nynorsk with a learning rate decay down to zero. The model is a base checkpoint and is not instruction-tuned (it is not a chat model), so it is designed for text continuation and research tasks. It is ideal for developers and researchers working on Norwegian text processing.
Key features
- Built on the Qwen3 architecture with 36 layers and a hidden state size of 2176.
- The model size is 2.12B parameters with a 4096 token context length.
- Underwent an annealing phase on high-quality Bokmål and Nynorsk texts.
- Trained using the Megatron-LM and Megatron-Bridge libraries for compatibility with transformers 4.x.
What you can do
- Use the model as a base for further continual pretraining for specific Norwegian tasks.
- Apply it to text autocomplete and content generation tasks in Norwegian.
- Integrate it into natural language processing (NLP) research pipelines for Scandinavian languages.
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("BigFamTLLM/da-emb-anneal")
model = AutoModelForCausalLM.from_pretrained("BigFamTLLM/da-emb-anneal", dtype=torch.bfloat16)
inputs = tokenizer("Oslo er hovedstaden i", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))Read more
DetailsNorwegian tokens: 1.00B
Context length: 4,096 tokens
Training: 8 GH200 GPUs
Limitations: research model, may generate incorrect or undesirable text










Comments