Qwen-VL: Multimodal AI model for text and object recognition in images
2 min readupdated
Qwen-VL by Alibaba Cloud is a large vision-language model (LVLM) capable of taking images, text, and bounding boxes as input, and returning text and object coordinates. Built on the Qwen language series, the model supports multilingual dialogue, multi-image processing, and detailed comprehension of graphical data. It is a great fit for developers and researchers who need accurate analysis of documents, charts, and objects in photos with localization support.
Key features
- Support for images, text, and detectable bounding boxes on input and output.
- Increased resolution (448 pixels) for direct end-to-end analysis without image cropping.
- Ready-to-use Qwen-VL-Chat version available for interactive conversations.
- Top scores on Flickr30K, General VQA, and OCR benchmarks.
What you can do
- Recognize text in images, read documents, charts, and tables.
- Highlight objects in photos based on text descriptions with coordinate output.
- Conduct multilingual dialogue using visual content as context.
- Integrate the model into image processing pipelines via Transformers.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-VL", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL", device_map="cuda", trust_remote_code=True).eval()
query = tokenizer.from_list_format([
{'image': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'},
{'text': 'Generate the caption in English with grounding:'},
])
inputs = tokenizer(query, return_tensors='pt').to(model.device)
pred = model.generate(**inputs)
response = tokenizer.decode(pred.cpu()[0], skip_special_tokens=False)
print(response)
Read more
DetailsRequirements: Python 3.8+, PyTorch 1.12+ (2.0+ recommended), CUDA 11.4+ for GPU acceleration.
Benchmarks: SOTA in the Flickr30K category for zero-shot captioning and leading results in general VQA among models of a similar scale.
Training features: The model supports an increased resolution of 448 pixels and native fine-tuning mode.










Comments