TableMoE (Qwen2.5-VL-7B)
VLM with expert routing for table analysis
2 min readupdated
A vision-language model (VLM) from anonymous researchers built to answer questions about table images. It is based on Qwen/Qwen2.5-VL-7B-Instruct, extended with LoRA adapters (lightweight weight add-ons) and an MoE module (Mixture-of-Experts, an architecture with multiple specialized experts and a router). Instead of a single vision-to-text connector, it uses four experts tailored to different data formats: general, HTML, JSON, and code. Ideal for developers and researchers looking to extract facts, run calculations, and parse tables from screenshots and scans.
Key features
- The connector integrates a router and 4 experts in a fixed order:
0=general,1=html,2=json,3=code. - Joint training (SC-RPO method, epoch 1) updates the router, all four experts, and the LoRA weights simultaneously.
- Inference uses deterministic Top-2 routing and greedy decoding (
do_sample=False). - Variants based on
Qwen3-VL-8Band an epoch 2 checkpoint are also available.
What you can do
- Integrate into pipelines that extract data from scanned invoices, reports, and table screenshots with natural-language answers.
- Apply to tabular QA and fact-checking tasks (similar to the WTQ, TAT-QA, and TabFact benchmarks).
- Compare the epoch 1 Qwen2.5-VL-7B checkpoint against the epoch 2 checkpoint or the Qwen3-VL-8B variant.
Example
export PYTHONPATH=code
python -m tablemoe.infer \
--release-dir . \
--base-model Qwen/Qwen2.5-VL-7B-Instruct \
--image /path/to/table.png \
--question "Your question about the table." \
--device cuda --max-new-tokens 1024 --output generation.json
Read more
DetailsBenchmarks (73.39 average): TabMWP — 88.47, TabFact — 80.85, InfoTabs — 72.33, HiTab — 69.67, TAT-QA — 64.90, WTQ — 64.11.
Loading: standard
AutoModel.from_pretrained is not supported; you need the loader from the code/ folder and a separately downloaded Qwen/Qwen2.5-VL-7B-Instruct base model.Limitations: anonymous checkpoint released for peer review (submission #15103), and the base model revision in the config is marked as
unverified.









Comments