NextLat-shooting-star
24 transformer checkpoints for graph pathfinding
3 min readupdated
A collection of 24 trained checkpoints of a compact transformer (91 MB each) built on NextLat (next-latent prediction, a method that predicts a model's upcoming hidden states, introduced by Teoh et al., 2025). Created by giovannimonea, these models tackle a synthetic algorithmic pathfinding task on "shooting star" graphs: given a list of 28 edges, the model must traverse an 8-node chain to a central hub, pick the right branch out of 5 rays, and reach the target node. Unlike standard next-token prediction, NextLat trains the model to forecast its own latent representations, helping overcome the well-known issue of guessing paths at branching points. Out of 24 initialization variants (seeds), 22 solve the task at over 97% accuracy while 2 fail—giving researchers a clean testbed for studying training stability and internal representations in algorithmic models.
Key features
- NextLat architecture: 12 layers, 6 attention heads, 384 hidden dimension, no bias terms, a 106-token vocabulary, and a 101-token context window.
- Strong accuracy across most seeds: 22 of the 24 models reach 97.79–99.98% accuracy with sampling and up to 100% with greedy decoding.
- Clear failure cases for comparison: seed 1234 completes the initial chain but picks the right ray only ~29% of the time, while seed 1254 scores 0% exact-match accuracy, making both useful for failure analysis.
- Full training logs: each seed ships with a
config.yamlfile,safetensorsweights, and accuracy evaluations recorded every 1,000 steps intraining_evals.json.
What you can do
- Run mechanistic interpretability experiments by comparing internal representations between successful seeds (such as 1239 or 1241) and failed ones (1234 and 1254).
- Analyze grokking and convergence dynamics on algorithmic graph tasks using the included 40,000-step training logs.
- Reproduce experiments from the NextLat paper on the
giovannimonea/shooting-star-graph-valvalidation dataset.
Example
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from models.model_nextlat import NextLat, NextLatConfig
cfg = NextLatConfig(n_layer=12, n_head=6, n_embd=384, block_size=101, bias=False, vocab_size=106, mtp_horizon=3)
model = NextLat(cfg)
model.model.load_state_dict(load_file(hf_hub_download("giovannimonea/nextlat-shooting-star", "seed1235/model.safetensors")))
model.model.eval()
Read more
DetailsMetrics on 20,000 test graphs: 22 of 24 seeds score 97.79% to 99.98% (sampled, temperature=1) and up to 100.00% (greedy, seeds 1239, 1241, 1247); seed 1234 hits 28.94% (greedy), and seed 1254 gets 0.00%.
Checkpoint size: 91 MB in fp32 (excluding optimizer states), runnable on a single GPU.
Training setup: 40,000 steps with batch sizes of 512 freshly generated graphs in bf16 mixed precision (AdamW, lr 5e-4, horizon 3, λ_mse=1, λ_kl=1).
Requirement: running the models requires cloning the official
JaydenTeoh/NextLat repository (commit 3770be6).









Comments