DON IT Fashion CIR Hybrid v0.1
image-and-text clothing search
3 min readupdated
A research hybrid system from the DON IT team built for Composed Image Retrieval (searching images using a "reference photo + text modification" pair). You upload a photo of a clothing item and add a refinement like "the same dress, but with a higher neckline," and the system ranks matching items from a catalog. Under the hood, it combines the Qwen3-VL-Embedding-8B and Marqo FashionSigLIP base models with custom adapters from the author and the external CSMCIR model. Unlike single-embedding setups, it blends rankings via RRF (Reciprocal Rank Fusion — a method for merging multiple result lists). Useful for developers building fashion search engines and visual product discovery tools for e-commerce.
Key features
- Three-branch hybrid architecture: pairs Qwen3-VL-Embedding-8B with a fashion LoRA (a lightweight add-on on top of the base model), Marqo FashionSigLIP with a distilled 768-dimensional head, and an external CSMCIR checkpoint.
- Distillation on real pairs: a new residual head trained on 12,917 user queries from the FashionIQ training split.
- Fixed fusion without test-set tuning: branch weights (70% Qwen and 30% SigLIP) and the RRF merge with CSMCIR (50/50 split with a constant of 60) were not tuned on the public validation set.
- Transparent research reporting: the authors explicitly note that the gain over the previous hybrid version (+0.167 percentage points) is not statistically significant, and scores on the shirts category dipped slightly.
What you can do
- Build a refinement-based clothing catalog search: use the adapters and scoring code to rank dresses, shirts, and tops for queries like "the same style, but in a different color or cut."
- Test a lighter setup without CSMCIR: run the Qwen3-VL + trained student module branch standalone if integrating the third-party CSMCIR checkpoint isn't an option.
Example
Sample workflow from the model card: you pass in a photo of a dress alongside the text instruction "the same dress but with a higher neckline". The system filters out the source image from the results and ranks catalog items within the same clothing category, pushing higher-neck styles to the top.
Read more
DetailsOn the FashionIQ public validation set (6,016 queries, 15,536 images), the hybrid achieved 59.5716% R@10 and 79.4064% R@50 (56.3506% R@10 without CSMCIR).
Training the student head took 600 steps on an A10G GPU (with the step-200 checkpoint selected).
Limitations: the repository currently includes only the adapter weights and feature-scoring code, with no ready-to-run end-to-end inference pipeline for raw images.
Full reproduction requires third-party weights for Qwen3-VL-Embedding-8B, Marqo FashionSigLIP, and CSMCIR; no license is specified yet.








Comments