DM-GDNGQA-9.3B: A hybrid language model with FP8 Delta-Matching attention
2 min readupdated
The DM-GDNGQA-9.3B-LR1.2e-3-DM model, featuring 9.27 billion parameters, is an experimental development with a hybrid architecture combining GatedDeltaNet layers and GQA (Grouped-Query Attention, a variant of the attention mechanism with a reduced number of key and value heads). It was trained from scratch on 30 billion tokens using the Delta-Matching method for FP8 attention. Created as part of research into training precision, this model delivers results fully comparable to standard FP32/BF16 training. It will be useful for language model architecture researchers and developers exploring efficient quantization methods.
Key features
- Hybrid architecture: GatedDeltaNet layers are combined with GQA (GatedDeltaNet, GatedDeltaNet, GatedDeltaNet, GQA, repeating 9 times).
- Trained on 30B tokens: Pretrained with a peak learning rate of 1.2e-3 on the Nemotron-CC dataset.
- Exported to bf16: Weights are saved in bf16 safetensors format with support for standard SDPA attention.
- High metrics: The model achieves a val CE of 1.2985 and a score of 44.11 on the MMLU benchmark.
What you can do
- Run text generation: Use the model for text completion and generation via standard Hugging Face tools.
- Analyze the architecture: Explore the features of the hybrid GatedDeltaNet layers and short convolution in flash-linear-attention.
- Fine-tune: Perform fine-tuning using the standard
Trainerclass from the transformers library.
Example
Example code for running model inference in Python:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "haok1402/DM-GDNGQA-9.3B-LR1.2e-3-DM"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, trust_remote_code=True).cuda()
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=32)[0], skip_special_tokens=True))Read more
DetailsMetrics (seed 11785): val CE 1.2985, RULER 62.84, MMLU 0.4411, TriviaQA 0.2564.
Hardware requirements: Weights require 18.6 GB of VRAM in bf16.
Limitations: Batched prompts must have the same length due to the design of GatedDeltaNet layers without padding masking.
Stack: torch, transformers >= 5, flash-linear-attention 0.5.2.










Comments