Spectator: A world model for predicting brain dynamics during movie watching
3 min readupdated
Spectator is an exploratory world model from the paper State of Mind: Learning and Evaluating World Models of Naturalistic Brain Dynamics, which forecasts human brain activity while watching movies. It takes as input recent fMRI (functional magnetic resonance imaging, divided into 1000 Schaefer-1000 regions) recordings and video stream features from the TRIBE v2 model, forming a latent state of 2048 numbers and rolling out predictions forward. Under the hood, it relies on a linear encoder trained using the JEPA principle (latent-space prediction without raw data reconstruction) and frozen, along with a transition matrix calculated in closed analytical form. Unlike approaches like TRIBE v2 that rely solely on stimuli, Spectator accounts for the current state of the viewer and noticeably outperforms competitors on the BrainStateBench benchmark. The release will be useful for neuroscientists and computational neuroscience researchers working with the Algonauts 2025 and HCP datasets.
Key features
- Hybrid forecasting: combines the viewer's fMRI history (7.5, 120, and 180-second options) with movie context features from TRIBE v2.
- Analytical solution: the transition matrix to the next state is calculated in closed form, and a linear probe is used for decoding into 1000 brain regions.
- Ready checkpoints and metrics: weights for the Algonauts 2025 and HCP datasets (184 participants, 7T fMRI) are published in the repository, along with saved BrainStateBench results.
- CC BY-NC 4.0 license: the model and used TRIBE v2 features are available for non-commercial research.
What you can do
- Reproduce experiments: run a 40-step fMRI rollout and recalculate linear probes on the Algonauts 2025 (Friends S6 and Movie10 films) or HCP datasets.
- Compare your architectures: upload custom models to the ready-made
benchmark-resultsstorage and get paired 95% confidence intervals against Spectator. - Fine-tune the transition matrix: use the frozen encoder (
encoder.pt) as a starting point for analytical fitting on new recordings.
Example
folder = "outputs/spectator-v1/algonauts/model"
cfg = load_config(f"{folder}/config.yaml")
model, _ = load_world_model(cfg, f"{folder}/best.pt", "cpu")
ref = torch.load(f"{folder}/reference-rollout.pt", weights_only=True)
with torch.no_grad():
states = model.rollout(ref["brain"], ref["stimuli"], horizon_trs=40)
Read more
DetailsOn the Algonauts 2025 test (180 s history): Forecast — 16.0%, State — 5.5%, Updating — 20.3%, Pearson correlation r = 0.349 (versus 11.6%, 2.6%, 12.7% and r = 0.294 for Linear forecaster).
On HCP (184 viewers): Forecast 6.6% on new content and 9.6% on new viewers, State 6.2%, Updating 15.7% and 18.2%.
File sizes: the main Algonauts checkpoint weighs 181 MB, the HCP version is 253 MB, and the benchmark storage is 503 MB.
Limitations: the checkpoints do not contain optimizer state (gradient fine-tuning cannot be continued), and the fMRI recordings themselves are not included in the repository.










Comments