1pp-0.58b-v2-spp-base
base checkpoint for studying reflection during pretraining
2 min readupdated
A 0.58-billion-parameter research language model from the 1PP project (EPFL DLAB), built as an SPP{T0} baseline (a method that injects first-person reflections into training texts to improve safety). Unlike standard pretraining on the DCLM-edu corpus, 7.73 million documents here include reasoning blocks appended after the special token, while a custom attention mask prevents subsequent text from "seeing" the injected reflection. This is a base pretraining artifact intended for researchers and engineers studying scaling and data filtering, not a ready-to-use chat assistant.
Key features
- Llama-style architecture: 24 layers, hidden dimension of 1,152, FFN size of 4,608 (SwiGLU), RMSNorm, and a 4,096-token context window.
- Trained on 23.29B tokens from the DCLM-edu corpus using the Muon optimizer for matrix parameters and Adam for the rest.
- SmolLM2-based tokenizer (49,152 tokens) extended with
<|pad|>(49152) and(49153) special tokens. - Outputs either standard text or first-person reflection (up to 256 tokens) when the
marker is appended after a document.
What you can do
- Compare behavior and generation quality against the paired raw checkpoint 1pp-0.58b-v2-raw-base in safety experiments.
- Study how injecting reflections into pretraining data impacts downstream SFT (supervised fine-tuning) and model alignment.
- Test reflection generation on documents by passing text followed by the closing
token.
Example
To trigger first-person reflection generation, append the special token to the end of the document text (note that standard causal attention in Hugging Face allows text after the reflection to attend to it, even though this was masked out during training). Dialogue formatting uses the ChatML template without a system prompt:<|im_start|>user
{message}<|im_end|>
<|im_start|>assistant
{reply}<|im_end|>
Read more
DetailsValidation loss (across 2,433 held-out documents): 2.615 on document text, 2.770 on assistant turns, and 2.790 on user turns.
Training: 11,106 steps (global batch size of 512 × 4,096 tokens) in bf16 precision with cross-document masking.
Limitations: this is a base research checkpoint, not a general-purpose assistant; there is no end-of-reflection marker.










Comments