FlagRelease/Hy4-W8A8-iluvatar-FlagOS
Tencent Hunyuan multimodal model for Iluvatar chips
2 min readupdated
Tencent Hunyuan has officially open-sourced its Hy4 multimodal model, and the FlagOS community has performed cross-hardware adaptation and prepared a W8A8 quantized version. The model is tailored for real-world performance scenarios, supporting visual understanding, computer use, complex task execution, and long-text processing. Built on the FlagOS 2.2 stack, the version is optimized to run on alternative hardware platforms while delivering high inference speed. The release is intended for developers and engineers deploying large multimodal models on heterogeneous hardware.
Key features
- Cross-hardware adaptation: Official integration of the Hy4 model into the FlagOS ecosystem with multi-vendor support.
- Performance optimization: Using the vLLM combination and the FlagGems operator library to accelerate computations.
- W8A8 quantization: Lower memory requirements and higher throughput while preserving generation quality.
- Multimodality: Full support for text and visual tasks, including the analysis of complex charts and interfaces.
What you can do
- Deploy an inference server: Launch a distributed cluster based on 16 Iluvatar BI-V150 cards using a ready-made Docker image.
- Integrate into chatbots: Connect the model to external client applications and services via the vLLM OpenAI-compatible API.
- Process images: Send multimodal prompts with graphical data for visual analysis and image-based answers.
Example
curl http://localhost:8400/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "hy4",
"messages": [{"role": "user", "content": "中国的首都是哪个城市?"}],
"max_tokens": 64,
"temperature": 0
}'Read more
Details• Architecture: HyV4VLForConditionalGeneration (246 shards, 793 GB)
• Hardware: 2 nodes × 16 Iluvatar BI-V150 cards (32 GB)
• Configuration: Tensor Parallelism 8, Pipeline Parallelism 4
• Limitations: it is critically important to use the PIECEWISE graph mode and not clear the /root/.triton cache










Comments