CS-CLIP - An improved CLIP for precise text-image matching
2 min readupdated
The CS-CLIP model is built on OpenAI CLIP ViT-B/32 and fine-tuned on a dataset featuring so-called hard negatives (difficult negative examples based on scene graphs). Standard CLIP models often confuse word order or object relationships, mixing up phrases like "a person on a horse" and "a horse on a person," for instance. This fine-tune solves this problem, providing more reliable compositional reasoning for text-to-image retrieval and zero-shot classification. It will be useful for multimodal system developers and computer vision researchers.
Key features
- Compositional reasoning: handles complex text descriptions and object relationships better.
- Architecture compatibility: retains the original OpenAI CLIP ViT-B/32 structure, stored in fp32.
- COCO training: trained on nearly 2 million caption pairs and hard negatives.
- Dual format: weights available for both the OpenCLIP library and Transformers.
What you can do
- Enhanced search: integrate into image search engines for precise matching of queries with complex context.
- Classification: use for zero-shot image classification without additional training for specific classes.
- Multimodal pipelines: apply alongside other computer vision models where understanding text and frame structure is critical.
Example
Running the model using the OpenCLIP library:
import torch, open_clip
from PIL import Image
model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:juns/CS-CLIP")
tokenizer = open_clip.get_tokenizer("hf-hub:juns/CS-CLIP")
model.eval()
image = preprocess(Image.open("example.jpg")).unsqueeze(0)
text = tokenizer(["a man riding a brown horse", "a brown man riding a horse"])
with torch.no_grad():
sims = model.encode_image(image, normalize=True) @ model.encode_text(text, normalize=True).T
Read more
DetailsBenchmarks: SugarCrepe Avg 88.59, VG-Rel 85.96, GQA two-choice 71.27
Architecture: OpenAI CLIP ViT-B/32 (QuickGELU), fp32 weights
Training: AdamW, lr 2e-6, batch size 128, 2 epochs on COCO train2014
Limitations: inherits general biases of pretrained CLIP and was fine-tuned exclusively on COCO data








Comments