SIMB: A Cross-Modal Speech and Vision AI Model
1 min readupdated
Developed by the KU-AGI team, the SIMB model combines the visual expertise of Janus-Pro-7B with a speech expert based on Qwen2.5-3B. It handles cross-modal synthesis tasks including speech-to-image generation, image-to-speech conversion, and spoken visual question answering (spoken VQA). The base visual expert remains frozen and identical to the original, while specialized bridge modules connect the different modalities. This new release is ideal for researchers and developers working on multimodal systems.
Key features
- Support for generating images from audio recordings
- Reverse conversion of images into speech tokens
- Ability to ask questions about images and receive spoken answers
- Fully standalone repository that requires no third-party weight downloads
What you can do
- Generate an image from an audio prompt using the built-in code
- Add voice output to visual content within multimodal pipelines
- Build an experimental voice assistant with image comprehension
Example
from simb import SIMB # pip install -e . in [https://github.com/KU-AGI/SIMB](https://github.com/KU-AGI/SIMB)
model = SIMB.from_pretrained("KU-AGI/SIMB")
image, _ = model.generate_image("prompt.wav", seed=0)
Read more
DetailsTotal parameters: 11.8B
Weight format: bf16 (for main components)
Base models: Janus-Pro-7B, Qwen2.5-3B, DualCodec, w2v-BERT 2.0
License: combined (requires compliance with DeepSeek, Qwen, and MIT terms)










Comments