MobileCLIP2-S4 (ONNX)
Apple model export for fast image recognition
2 min readupdated
This model is a conversion of Apple's original multimodal MobileCLIP2-S4 development into the ONNX format. It performs zero-shot image classification (classifying images without prior training on those classes) and processes text and visual prompts separately. The output matches probabilities for given text labels and an image using ONNX Runtime. Based on the FastViT architecture with re-parameterized branches for inference, it was released by author AX1Y2JP. Compared to the original versions, this release features optimized graphs, fp16 and int8 weights for faster performance, and a adjusted tokenizer to preserve recognition accuracy.
Key features
- Exported to ONNX format (opset 23) for use with ONNX Runtime 1.22+
- Separate graphs for visual and text encoders
- FastViT encoder re-parameterization with merged branches for fast inference
- Optimized options available with fp16 weights and 8-bit quantization (MatMulNBits)
- Preserved logit_scale parameter in model metadata
What you can do
- Integrate the model into Python applications using onnxruntime for image classification
- Perform image classification using custom text labels without fine-tuning
- Run inference on GPU (CUDAExecutionProvider) or CPU to speed up processing
Example
import numpy as np
import onnxruntime as ort
from PIL import Image
from tokenizers import Tokenizer
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
vision = ort.InferenceSession("onnx/vision_model_fp16.onnx", providers=providers)
text = ort.InferenceSession("onnx/text_model_fp16.onnx", providers=providers)
scale = float(vision.get_modelmeta().custom_metadata_map["logit_scale"])
img = Image.open("cat.jpg").convert("RGB")
w, h = img.size
r = 256 / min(w, h)
img = img.resize((round(w * r), round(h * r)), Image.BILINEAR)
w, h = img.size
l, t = (w - 256) // 2, (h - 256) // 2
img = img.crop((l, t, l + 256, t + 256))
pixel_values = (np.asarray(img, np.float32) / 255).transpose(2, 0, 1)
pixel_values = (pixel_values - np.array([0.48145466, 0.4578275, 0.40821073], np.float32)[:, None, None]) / np.array([0.26862954, 0.26130258, 0.27577711], np.float32)[:, None, None]
pixel_values = pixel_values[None]
tok = Tokenizer.from_file("tokenizer.json")
labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
input_ids = np.array([e.ids for e in tok.encode_batch(labels)], np.int64)
img_emb = vision.run(None, {"pixel_values": pixel_values})[0]
txt_emb = text.run(None, {"input_ids": input_ids})[0]
img_emb /= np.linalg.norm(img_emb, axis=-1, keepdims=True)
txt_emb /= np.linalg.norm(txt_emb, axis=-1, keepdims=True)
logits = scale * img_emb @ txt_emb.T
probs = np.exp(logits - logits.max(-1, keepdims=True))
probs /= probs.sum(-1, keepdims=True)
print(dict(zip(labels, probs[0].round(3))))
Read more
DetailsBase model: apple/MobileCLIP2-S4
Requirements: ONNX Runtime 1.22+
Tokenizer features: tokenizer.json padded with zeros to match open_clip instead of the original HF tokenizer










Comments