RetCast-JEPA
ultra-lightweight foundation models for time series forecasting
2 min readupdated
A family of tiny models ranging from 2.05M to 9.19M parameters, built from scratch on the RetNet2.0 architecture and the JEPA approach (Joint Embedding Predictive Architecture, where the model predicts future semantics in latent space rather than noisy raw values). It combines a delta-rule rewritable memory with embedding prediction, outperforming much heavier transformers across several tasks. The models were trained on diverse time series datasets in a zero-shot setup without seeing target test tasks beforehand. It is a great fit for researchers and engineers who need fast, accurate time series forecasting on low-end hardware.
Key features
- Ultra-lightweight footprint: variants from 2M to 9M parameters take up just 8 to 35 MB in memory and require minimal compute.
- High speed: runs inference significantly faster than bulky alternatives like Google TimesFM 200M.
- RetNet2.0 + JEPA architecture: uses delta-rule memory and pattern prediction instead of step-by-step noisy data generation.
- High accuracy: achieves low MSE and MAE scores on benchmarks while keeping VRAM usage minimal.
What you can do
- Forecast time series: generate point and quantile forecasts for numerical sequences using ready-made model variants (`2m`, `5m`, `9m`).
- Run recurrent rollouts: generate future states in latent space with constant $O(1)$ memory consumption.
- Test architectures: explore model components (including RetNet and transformer ablations) for research experiments.
- Integrate into pipelines: embed the models into PyTorch data analysis pipelines for GPU execution.
Example
import torch
from retcast import RetCastJEPA
# Загружаем модель RetCast-JEPA 2M с HuggingFace Hub
model = RetCastJEPA.from_pretrained("starkprateek1/retcast-jepa", variant="2m", device="cuda")
# Подготовка контекста (B, L)
context = torch.randn(4, 512, device="cuda")
with torch.no_grad():
out = model(context, horizon_len=192, mode="chunkwise")
point = out["point"] # Точечный прогноз (4, 192)
quantiles = out["quantiles"] # Квантили прогноза
Read more
DetailsBenchmarks: RetCast-JEPA 2M achieves a Mean MSE of 0.3706 compared to 0.3878 for Google TimesFM 200M.
Hardware requirements: the 2M variant uses only 22.0 MB of VRAM during inference.
License: MIT.










Comments