tell-fit-clip
Checking headline-to-image alignment in the browser
2 min readupdated
This model verifies whether a text headline accurately describes an attached photo and outputs a mismatch probability. Under the hood, it pairs the openai/clip-vit-base-patch32 image encoder with a multilingual DistilBERT text encoder from sentence-transformers, exported to ONNX to run client-side via ONNX Runtime Web. Unlike standard CLIP models, it uses 8-bit weight quantization (INT8) for compact browser storage and built-in logistic calibration for cosine similarity. It is useful for developers building news aggregators, content moderation tools, and web apps that need to filter out irrelevant image captions without adding server load.
Key features
- Supports over 50 languages (including English and Hindi) using a multilingual DistilBERT encoder and a WordPiece vocabulary.
- Runs inference completely client-side in the browser via ONNX Runtime Web, with no need to upload photos to a backend.
- Stores weights compactly in INT8 format (with per-output-channel scaling) and dequantizes them back to float32 on load.
- Provides calibrated scoring by passing the embedding cosine similarity through a logistic curve from
calibration.jsonto estimate error probability.
What you can do
- Build automated headline-to-image checks directly into a news editor or web portal pre-moderation system.
- Run client-side screening for clickbait or mismatched photo captions right in the user's browser with ONNX Runtime Web.
Example
The model card outlines the inference pipeline: a [n, 3, 224, 224] image tensor goes into fit_image_int8.onnx, while tokenized text (input_ids, attention_mask) feeds into fit_text_int8.onnx. Both models output normalized [n, 512] embeddings, and their cosine similarity is mapped to a headline mismatch probability using the parameters in calibration.json.
Read more
DetailsOn the Tell benchmark (50 photos paired with matching and random headlines), the 8-bit ONNX version matches the original PyTorch ranking in all 50 cases, with a maximum cosine similarity difference of 0.011.
It achieves a ROC AUC of 0.940 compared to 0.942 for the full-precision models.
Limitation: the model only evaluates visual alignment between the scene and the words, so it cannot catch plausible but factually wrong headlines (such as an incorrect location or event date).
Component licenses: MIT for
openai/clip-vit-base-patch32 and Apache 2.0 for clip-ViT-B-32-multilingual-v1.









Comments