Duo data-fraction scaling study: diffusion language model checkpoints
2 min readupdated
The repository contains intermediate and final checkpoints (model states) for five discrete diffusion Duo language models with a backbone size of around 169 million parameters. The models were trained on nested subsets of the OpenWebText dataset (ranging from 10% to 100% of the data) with a weight decay of 0.1 to study scaling laws. This set is useful for researchers studying text generation architectures and diffusion models to analyze training trajectories.
Key features
- Five model variants trained on 10%, 25%, 50%, 75%, and 100% of training blocks
- DiT architecture with a 1024-token context and GPT-2 tokenizer
- Uses the duo_base algorithm with a log-linear noise schedule
- Full checkpoints on a fixed step grid for training trajectory analysis
What you can do
- Analyze scaling laws and weight changes across different training stages
- Load checkpoints to continue training or extract backbone weights
- Explore validation metrics and weight norms from the included JSONL files
Example
import torch
ck = torch.load("frac_0.10/checkpoints/step_0475000.ckpt", map_location="cpu", weights_only=False)
raw = {k: v for k, v in ck["state_dict"].items() if k.startswith("backbone.")}
print(ck["global_step"], ck["scaling_meta"]["clock"]["train_s"] / 3600, "training hours")Read more
Details• Data: OpenWebText, 8.7M blocks for the full version
• Training: AdamW (lr 3e-4), bf16 mixed precision
• Metrics: Validation NELBO from 3.3461 to 3.3763 nats/token
• Limitations: Training time mixes H100 and H200 GPUs










Comments