LFM3-1.7B-Flash
LLM inference acceleration
2 min readupdated
The model adds speculative decoding (a token generation speedup method) to the base LiquidAI/LFM3-1.7B LLM using a specialized draft head architecture and suffix search. It delivers responses 2–4x faster without losing accuracy or deviating from the original model's outputs. The solution integrates via the standard transformers library and is well-suited for local assistants, data parsing, JSON handling, and background tasks.
Key features
- Speculative decoding: uses a 2-layer Osprey head and suffix/prompt search to speed up generation.
- Flexible configuration: supports adjusting search tree size, block parameters, and tokenizer sampling.
- Transformers compatibility: loads via the standard
AutoModelForCausalLM.from_pretrained()method with an activation flag. - Multilingual: maintains high accuracy when handling requests in Russian and other languages.
What you can do
- Local RAG search: set up a fast question-answering system over your own text notes and documents.
- Structure extraction: parse unstructured notes and messages into ready-to-use JSON objects.
- Bot integration: use the model in Telegram bots for instant replies and short summaries.
- Web scraping: automate data extraction from HTML product cards and news digests.
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "cybertruck32489/lfm3-1.7b-flash"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.float16,
device_map="auto",
trust_remote_code=True,
).eval()
model.prepare_flash_inference()
inputs = tokenizer("Explain TCP vs QUIC", return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=128, use_dflash=True)
print(tokenizer.decode(output_ids[0]))Read more
Details• Base model: LiquidAI/LFM3-1.7B (1.87 billion parameters)
• Performance: up to 4.56x speedup on JSON extraction tasks on an NVIDIA Tesla T4 GPU
• License: LFM Open License v1.0
• Training highlights: head-reordered 48K token vocabulary support










Comments