Pascal-15M-v1.0
a micro-transformer for numerical sequence continuation
2 min readupdated
Pascal-15M-v1.0 from AtomixLabs is a compact 17.7M-parameter language model that can't write prose or code, but excels at spotting patterns in numbers and continuing sequences. Under the hood, it uses a standard 12-layer LLaMA architecture trained on roughly 364M tokens of numerical data. Unlike standard LLMs that glue digits into arbitrary chunks via BPE tokenization, it uses an exact 32-token vocabulary—one token per digit, separator, and sign. Researchers and developers can use it as a lightweight base for fine-tuning on specific numerical tasks.
Key features
- 32-token vocabulary: includes only digits 0–9, comma, minus sign, period, slash (
/), and special tokens, eliminating rounding errors and tokenization artifacts. - Handles diverse sequence types: uncovers rules in arithmetic and geometric progressions, fractional approximations (such as Fibonacci numbers), alternating streams, and oscillators.
- Native LLaMA architecture: runs out of the box in PyTorch and Transformers with no custom CUDA kernels or
trust_remote_code=Truerequired. - Controlled output formatting: the tiny vocabulary makes it easy to constrain generation via logit masking, blocking invalid syntax like
/0or duplicate signs.
What you can do
- Telemetry and sensors: fine-tune the checkpoint on sensor readings or monitoring logs to forecast upcoming values.
- Financial ticks: adapt the model to analyze discrete price levels, spreads, and rate changes.
- Dynamic system simulation: use it as a foundation for kinematics, coupled oscillations, or empirical symbolic regression.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("AtomixLabs/Pascal-15M-v1.0", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("AtomixLabs/Pascal-15M-v1.0")
inputs = tokenizer("1, 1/2, 2/3, 3/5, 5/8,", return_tensors="pt").to(model.device)
sep_id = tokenizer.convert_tokens_to_ids(",")
out = model.generate(**inputs, max_new_tokens=24, do_sample=False, eos_token_id=sep_id)
print(tokenizer.decode(out[0], skip_special_tokens=True)) # 1,1/2,2/3,3/5,5/8,8/13
Read more
DetailsParameters: 17,716,608 (hidden_size 384, 12 layers, 12 attention heads, 512-token context).
Note: the model wasn't trained with an
token, so stopping must be handled manually via commas or max_new_tokens.Limitations: doesn't understand natural text, is sensitive to whitespace and formatting, and tends to hallucinate spurious patterns when fed random noise.
License: Apache-2.0.










Comments