Tree-gram generators
a family of autoregressive text generators
2 min read
A collection of compact autoregressive models (which predict the next token from preceding ones) by hugs-from-a-25 for generating synthetic text corpora. Under the hood is the tree-gram architecture—a hierarchical structure that estimates the next token based on a window of n - 1 preceding tokens. The repository includes 30 model variants with varying context lengths (1 to 31 tokens) and layer widths d (64 to 1024), trained on three well-known datasets: TinyStories, WikiText-103, and FineWeb-Edu. It is useful for researchers and developers looking to quickly sample synthetic text, study context scaling, and experiment with lightweight generative models without the overhead of full-scale LLMs.
Key features
- 30 pre-trained variants: 10 configurations for each of the three domains (TinyStories, WikiText-103, and FineWeb-Edu) spanning different context depths and dimensions.
- Minimal dependencies: running the
modeling_treegram.pyscript requires onlytorchandsafetensors. - Classic bigram support: alongside the neural tree-gram models, the suite includes frequency-based
bigrammodels incounts.npzformat with uniform distribution fallback. - Full metric transparency: the model card reports exact entropy and validation loss in nats per token for every checkpoint.
What you can do
- Generate synthetic datasets: produce text in the style of children's stories, encyclopedia articles, or educational content to train and test other models.
- Benchmark context and width scaling: use the provided checkpoints to test how varying context length (1 to 31 tokens) and width
d(64 to 1024) affect sampling quality and entropy. - Integrate into research pipelines: call the
next_token_logitsfunction to extract next-token logits and analyze probability distributions in your own PyTorch scripts.
Example
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("hugs-from-a-25/tree-gram-generators", allow_patterns=["modeling_treegram.py", "tinystories/n8_d1024/*", "tokenizers/*"])
sys.path.insert(0, path)
from modeling_treegram import load_generator, sample
model, cfg = load_generator(f"{path}/tinystories/n8_d1024")
tokens = sample(model, n_tokens=200, batch=4, eos_token_id=cfg["eos_token_id"])
Read more
DetailsWeights are stored in
model.safetensors format (float32), with config.json and generation.json included for each model.Best validation loss per group: TinyStories (
n32_d1024) — 1.7407, WikiText-103 (n32_d1024) — 3.9298, FineWeb-Edu (n16_d1024) — 4.2983 nats/token.VRAM and hardware requirements are not specified in the model card.
Licenses depend on the corpus: CDLA-Sharing-1.0 (TinyStories), CC BY-SA 3.0 (WikiText-103), and ODC-By 1.0 (FineWeb-Edu).










Comments