Mini CLM: A compact contrastive language model without text generation
2 min read
Mini CLM is an independent implementation of the contrastive language model concept, which evaluates response option probabilities in a single pass instead of token-by-token text generation. The model is built on a frozen Qwen/Qwen3-0.6B base, supplemented with two small trained heads (state head and action head) to match situations and options via cosine similarity. The project is designed for multiple-choice tasks and ranking evaluation on a laptop GPU. It will be useful for researchers and developers exploring alternative approaches to evaluating LLM (large language model) responses.
Key features
- Qwen3 base: Uses the compact
Qwen/Qwen3-0.6Bmodel as a frozen backbone. - Contrastive approach: Returns the probability for each option in a single forward pass without generating tokens.
- Trained heads: Contains a state head and an action head trained in three stages using InfoNCE loss.
- Specialization: Excels at semantic matching on datasets like ARC-Easy and CommonsenseQA, but does not perform step-by-step reasoning.
What you can do
- Perform multiple-choice tasks and rank predefined response options to questions.
- Test action-selection scenarios for agents (based on datasets like
neulab/adp-v2). - Run the ready evaluation script using the included
clm.pymodule and a demo example.
Example
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("ManiShekhar/mini-clm")
sys.path.insert(0, path)
import clm
scorer = clm.Scorer()
print(scorer.rank("Question: What is the capital of Japan?", ["Tokyo", "Osaka", "Paris"]))
Read more
Details• Requirements: Requires a CUDA GPU and local execution via
clm.py.• Training: 3 stages (pre, mid, post) with a learning rate ranging from 3e-4 to 5e-5.
• Limitations: The model cannot perform step-by-step reasoning and does not output numerical scores.










Comments