LiberalMind Video VAE
video variational autoencoder
2 min readupdated
This model from the LiberalMind research initiative handles video sequence compression and reconstruction, serving as a core component for robotics systems and world models. The architecture is trained on 256×256 pixel clips of 17 frames using a combination of frame-by-frame reconstruction, LPIPS perceptual similarity, and latent regularization. The checkpoint targets researchers in embodied AI and temporal visual data analysis. To run the model, you need to separately connect the architecture description file vae.py in addition to the weights.
Key features
- Video representation: Encoding and reconstructing short video sequences using a variational autoencoder (VAE).
- TPU training: Configuration optimized for Google Cloud TPU using the PyTorch/XLA stack and SPMD parallelism.
- Perceptual loss: Using the LPIPS (VGG) metric to evaluate visual similarity.
- Data format: Processing input tensors of dimension [B, 3, 17, 256, 256] in a precise BF16 grid.
What you can do
- Robotics research: Use the model as a foundation for extracting visual features in perception tasks for autonomous agents.
- Video sequence analysis: Apply it for temporal visual understanding tasks and creating compact latent representations.
- Run training on TPU: Deploy the fine-tuning pipeline on custom datasets using the provided instructions and scripts.
Example
Example code for loading the downloaded checkpoint and initializing the model:
import torch
from vae import VAE
checkpoint = torch.load(
"last.pt",
map_location="cpu",
weights_only=False,
)
model = VAE()
model.load_state_dict(checkpoint["model"])
model.eval()
print("Model loaded successfully")
print("Training step:", checkpoint["step"])
Read more
DetailsInput resolution: 256×256 pixels
Sequence length: 17 frames
Optimizer: AdamW (learning rate 1e-4)
Framework: PyTorch / PyTorch/XLA










Comments