SILY-1 Chat
A Compact MoE Chat Model
2 min readupdated
SILY-1 Chat is a mixture-of-experts (MoE, an architecture splitting computation across independent subnetworks) language model trained for conversational tasks. The model has 476.9 million total parameters but activates only 113.5 million per token. Built on top of SILY-1 Base, it was trained on nearly two million dialogues in just 53 hours using a single RTX 3060 graphics card. The model generates coherent, on-topic responses, though it can make mistakes in facts and arithmetic. It is well-suited for developers exploring the training of compact AI models.
Key features
- MoE architecture: 253 experts per bank with 4 active experts per token for efficient performance.
- Context up to 1,024 tokens and a 12,288-token tokenizer.
- Trained on 1.98 million chat rows (1.05 billion tokens) for a single epoch.
- Apache 2.0 license for free use.
What you can do
- Run a multi-session command-line chat using the
chat.pyscript. - Integrate the model into Python scripts to generate short answers and poems.
- Adjust sampling parameters (temperature, repetition penalties) for text experiments.
Example
import torch
from tokenizers import Tokenizer
import chat
from generate import SamplerConfig, special_ids
from modeling import load_model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model, config = load_model('model.pt', device)
tok = Tokenizer.from_file('tokenizer.json')
tok.encode_special_tokens = True
ids = chat.chat_ids(tok)
stop = ids['eass']
cfg = SamplerConfig(temperature=0.7, min_p=0.03, presence_penalty=0.3)
messages = [{'role': 'user', 'content': 'Write a short poem about the ocean.'}]
for r in chat.replies(model, tok, ids, messages, cfg, seeds=[0]):
print(r['reply'])
Read more
DetailsBenchmarks: Held-out loss dropped from 1.747 to 1.198 nats per token after fine-tuning.
Hardware: Training used a single RTX 3060 (12 GB) GPU, with peak VRAM usage reaching 9.6 GB.
Limitations: Facts and arithmetic are unreliable, and long contexts are truncated to 1,024 tokens.
Training details: AdamW optimizer, 3,866 steps, batch size of 512 rows, cosine learning rate decay.










Comments