AURA-1.3B
Audio-Driven Streaming Talking Head Avatar
2 min read
This model from the Kyutai lab turns a portrait image and an audio speech stream into a real-time talking-head video. It is built on a Wan2.1-T2V-1.3B architecture combined with Mimi audio modules (audio tokens) and distilled from a teacher model using DMD (Distribution Matching Distillation). The model outputs 576p video at around 20.7 frames per second on a single H100 GPU while strictly preserving the original face identity (CSIM). This release is intended for researchers and developers building interactive video avatars (under a CC BY-NC 4.0 license).
Key features
- Real-time streaming generation: outputs results chunk by chunk using a rolling KV cache.
- High face fidelity: achieves superior identity preservation (CSIM) compared to evaluated methods on the HALLO3 and CelebV-HQ benchmarks.
- Efficient distillation: a four-step student pipeline (4 denoising steps) accelerates generation by 23x compared to the base teacher model.
- Two weight formats: standalone files (*_merged) for quick deployment and base checkpoints (LoRA-only) for fine-tuning.
What you can do
- Animate portraits: generate realistic talking-head videos from a static photo and an audio speech recording.
- Create an interactive avatar: integrate the model into streaming systems or chatbots to generate video on the fly.
- Conduct research: use LoRA-only checkpoints and configuration files to customize distillation pipelines (ODE and DMD).
Example
git clone [https://github.com/kyutai-labs/aura](https://github.com/kyutai-labs/aura) && cd aura
uv sync
uv run generate.py --image portrait.jpg --audio speech.wav --output out.mp4Read more
DetailsHALLO3 benchmarks: FID 56.8, FVD 177.1, CSIM 0.84
Hardware requirements: one H100 GPU cited in the paper (~16-20+ fps)
Architecture: Block-wise causal over chunks of 4 latent frames (~0.64 s)
Limitations: CC BY-NC 4.0 license (non-commercial use only)










Comments