Transformer-1.3B EHWS: A 90% Compressed Sparse Language Model
2 min readupdated
The Transformer-1.3B-semi-structured-EHWS-90pct model is an experimental 1.3B-parameter language model with its linear layers artificially pruned down to 90% sparsity. It is built upon the base fla-hub/transformer-1.3B-100B model, using a two-phase ADMM method and C4 data calibration for pruning. The architecture features semi-structured sparsity: every row in the pruned linear layer retains an identical weight count (top 10%), while the embeddings and final lm_head remain dense. This model appeals to researchers and engineers exploring optimization methods, inference acceleration, and LLM compression techniques.
Key features
- Extreme compression: 90% sparsity applied to the linear layers of the model blocks.
- Balanced load: each row in the pruned layer maintains an equal number of non-zero weights (exactly 10%).
- Training method: two-phase optimization using ADMM and C4 calibration sequences.
- Integration: fully compatible with the
flash-linear-attention(fla) library and theAutoModelForCausalLMstandard.
What you can do
- Inference testing: run the model locally using the
flash-linear-attentionlibrary to experiment with sparse computations. - Quality analysis: compare accuracy and perplexity metrics between the sparse version and its dense base counterpart on text generation tasks.
- Optimization research: inspect the configuration files and
results.jsondata to reproduce the ADMM method on other architectures.
Example
import torch
from transformers import AutoModelForCausalLM
# Загрузка модели требует установленной библиотеки flash-linear-attention (import fla)
model_id = "ikimyaii/Transformer-1.3B-semi-structured-EHWS-90pct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
print(model)
Read more
Details• Perplexity (WikiText-2): Dense — 9.77, this model — 42.59
• Perplexity (C4): Dense — 14.65, this model — 38.86
• Average zero-shot accuracy (lm-evaluation-harness): 0.4390 (Dense) vs 0.3663 (this model)
• Training time: 2.76 hours on a single A100 GPU (40GB)
• Limitations: embeddings and lm_head remain dense, accuracy drops due to heavy pruning










Comments