Hy4-Flash-WV-v6
run a massive multimodal model on a single GPU
2 min readupdated
This new 2.50 bpw quantized build of the popular tencent/Hy4-flash multimodal model is tailored specifically for Blackwell GPUs (SM120 architecture). While the original checkpoint exceeds 500 GB and requires a server cluster, this version shrinks the weights down to 52.5 GB so it can run on a single consumer card like an RTX 5090 (with VRAM limits) or an RTX PRO 6000. It supports text, images, reasoning chains, and tool calls—making it ideal for developers and hobbyists who want to test a heavy modern architecture locally.
Key features
- Single-GPU execution: Compresses backbone weights to 52.51 GB while offloading heavy Engram data to host memory (mmap) without consuming VRAM.
- Speculative decoding: A built-in
DFly2(0.85B) draft model speeds up generation to 55–229 tokens per second. - Multimodal support: Full image processing capabilities powered by an integrated
SigLIP2 ViT. - OpenAI API compatibility: Ships with a built-in server for quick deployment and seamless web UI integration.
What you can do
- Spin up a local AI server on an RTX 5090 using a dedicated VRAM optimization flag.
- Test structured JSON outputs alongside streaming reasoning chains (
reasoning_effort). - Benchmark the quantized build against the original checkpoint using built-in memory and accuracy tests.
Example
pip install wheels/qfn_sm120_unified-22.3.0-cp313-cp313-linux_x86_64.whl
# Запуск на потребительской карте RTX 5090 с ограничением памяти:
python -m qfn_sm120_unified.serve --ckpt . --host 0.0.0.0 --port 8000 --force-32gb --inline-thinkRead more
Details• Backbone size: 52.51 GB (117.19 GB total on disk)
• Hardware requirements: 1x 96GB GPU or RTX 5090 (32 GB)
• Generation speed: up to 229.54 tokens/s (at batch size 32)
• License: Tencent Hy License










Comments