Lythri-Wink-2B-pt
a lightweight base model built on Gemma4
2 min readupdated
Lythri-Wink-2B-pt is a pre-trained base LLM (large language model) with 2.52B total parameters and 1.48B active parameters. It was created by structurally shearing Gemma4-E2B PT and recovering performance via teacher distillation (knowledge transfer from a larger model) over 813M tokens. Rather than working out of the box for chat, it is built for continued pre-training, fine-tuning (SFT), and research. It is a solid fit for developers and researchers looking for a compact architecture to customize.
Key features
- Built on the Gemma4-E2B PT architecture with FFN (feed-forward network) width reduced to 60%.
- Packs 2.52B total parameters with 1.48B active parameters across 31 decompression layers.
- Trained with an 8K token context length on a dataset of roughly 813M tokens.
- Released under the Apache License 2.0.
What you can do
- Run supervised fine-tuning to build a specialized assistant or chatbot.
- Perform continued pre-training tailored to a specific domain or task.
- Use it in academic research on structured pruning and model distillation.
- Build a production-ready text generation model on top of it using the inference code in the repository.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_path = "Lythri/Lythri-Wink-2B-pt"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
model_path,
dtype=torch.bfloat16,
device_map="auto",
)
prompt = "Once upon a time"
inputs = tokenizer(
prompt,
return_tensors="pt",
add_special_tokens=True,
).to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.8,
top_p=0.95,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))Read more
DetailsValidation metrics: Validation KL 0.3241, Student CE 2.552, Teacher CE 2.270.
Key benchmarks: ARC-E — 65.99, SocialIQA — 42.43.
Limitations: this is a base model not tuned for instruction following, and it inherits the limitations of the underlying architecture.










Comments