HGRN-1.3B-semi-structured-EHWS-80pct-seed2
Sparse Version of a Language Model
1 min readupdated
This model is an experimental version of the base fla-hub/hgrn-1.3B-100B, optimized using a two-phase ADMM algorithm to achieve 80% semi-structured sparsity. The model's linear layers retain an equal number of weights in each row, while the embeddings and final projection remain dense. The model is intended for researchers and developers exploring efficient architectures with linear attention and sparse weights. Running it requires the flash-linear-attention library.
Key features
- Semi-structured sparsity: 80% of the weights in linear layers are removed while maintaining a uniform structure across rows.
- ADMM optimization: Two-phase training with C4 data calibration to minimize accuracy loss.
- Efficiency: Training took just 3.71 hours on a single A100 40GB accelerator.
- Compatibility: Full integration with the standard
AutoModelForCausalLMinterface via theflalibrary.
What you can do
- Compression research: Test the behavior of sparse linear models in text generation tasks.
- Benchmarking: Compare performance and perplexity metrics against the base dense model on datasets like WikiText-2.
- Hardware experiments: Evaluate the potential for compute acceleration through uniform layer sparsity.
Example
from transformers import AutoModelForCausalLM
import fla
model = AutoModelForCausalLM.from_pretrained(
"ikimyaii/HGRN-1.3B-semi-structured-EHWS-80pct-seed2",
device_map="auto"
)
print(model)
Read more
DetailsPerplexity: WikiText-2 (42.49), C4 (29.72)
Average zero-shot (lm-evaluation-harness): 0.3821
Base model: fla-hub/hgrn-1.3B-100B
Parameters: 1.36 billion










Comments