TriCosmos-2.5-predict
multi-sensor road forecasting in adverse weather
3 min readupdated
A world model from SORA-Lab that predicts the next 2 seconds of vehicle motion across three sensors simultaneously: a standard RGB camera, a thermal imager, and a LiDAR (laser radar) depth map. It takes 5 observed frames at 10 Hz as input and outputs 20 future frames in fog, rain, snow, or at night. Built on the NVIDIA Cosmos-Predict2.5-2B base and fine-tuned via LoRA (low-rank adapters) on the DSERT-RoLL dataset, its core trick is packing all three streams into a single video frame arranged as a 2×2 grid. Useful for autonomous driving researchers and developers running simulations and planning in harsh weather.
Key features
- Unified 2×2 grid: a 1280×704 frame is split into four 640×352 tiles — RGB at top-left, thermal at top-right, LiDAR depth at bottom-left, and an empty black square at bottom-right.
- Higher LiDAR accuracy: joint training across all three modalities cuts depth prediction mean absolute error (MAE) down to 4.68 m — 2.1 m better than a comparable LiDAR-only LoRA model (6.81 m).
- Built for harsh weather: trained on 35 sequences from DSERT-RoLL covering dense fog (23 clips), heavy snowfall, rain, and night scenes.
- Ready-to-use merged checkpoint: the repository includes
tricosmos_ema_bf16.pt(3.9 GB, bf16 format) along with theexperiment.jsonconfigs needed to assemble the model with LoRA parameters.
What you can do
- Simulate adverse-weather driving scenes: plug it into
cosmos-predict2.5pipelines to generate 2-second multi-sensor scenarios in fog, snowfall, and nighttime conditions. - Benchmark multimodal world models: compare joint RGB, thermal, and depth predictions against single-sensor baselines on the DSERT-RoLL validation split.
- Fine-tune on custom sensor logs: use the published
train_tri_lora.yamlrecipe and 2×2 tile-packing layout to adapt Cosmos-Predict2.5-2B to your own automotive sensor recordings.
Example
import torch
sd = torch.load("tricosmos_ema_bf16.pt", map_location="cpu", weights_only=False)
sd = sd.get("model", sd) if isinstance(sd, dict) else sd
lora = [k for k in sd if "lora" in k.lower()]
print(len(sd), "tensors,", len(lora), "LoRA tensors")
Read more
DetailsMetrics (DSERT-RoLL validation, 2 s horizon): RGB PSNR — 21.71, Thermal PSNR — 24.61, LiDAR MAE — 4.68 m (vs. 5.71 m for simply repeating the last frame and 6.81 m for LiDAR-only LoRA).
Training: LoRA rank 16, alpha 32 on attention and MLP layers; 500 steps in bf16 on 93-frame clips; generation evaluated at 35 sampling steps and guidance 7.
Limitations: each sensor runs at half-tile resolution (640×352), simple last-frame repetition for LiDAR remains more accurate at close ranges under 10 m, and the authors plan to release the full inference code after paper peer review.
License: NVIDIA Open Model License (based on Cosmos-Predict2.5-2B).










Comments