ClosureRAG: Cross-encoders for evidence retrieval by regulations
2 min readupdated
ClosureRAG is a set of fine-tuned checkpoints based on Qwen3-Reranker-0.6B, designed for retrieval-augmented generation tasks where answering a question requires gathering an entire set of interrelated regulations or documents (known as proof closure). The models evaluate case-snippet pairs using yes/no logit differences at the last position. The repository includes two ranking levels (group retriever and context-dependent rescorer) as well as an ablation module, helping improve search accuracy in complex legal or regulatory databases.
Key features
- Qwen3-Reranker-0.6B base: all three models are fine-tuned from the efficient open-source 0.6B parameter ranker.
- Multi-level pipeline: split into Level-1 (group candidate coverage) and Level-2 (context-conditional rescoring).
- High FullCov and Recall metrics: the models show significant improvement in coverage on a test dataset of 1,160 complex questions.
- Complete code suite: the repository contains inference scripts, query preparation, and evaluation scripts.
What you can do
- Integrate into a RAG system: apply the two-level retriever in projects where answering questions requires collecting a complete set of interrelated documents or rules.
- Run inference: use the scripts from the repository to evaluate candidates on test pools using
scripts/run_inference.sh. - Score candidates in Python: use ready-made code examples to evaluate individual candidates via transformers on PyTorch.
Example
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from closurerag.fmt import Fmt, score
path = "models/l1_coverage_ranker"
tok = AutoTokenizer.from_pretrained(path, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).cuda().eval()
F = Fmt(tok, 768)
case = F.case(question_text)
seqs = [F.encode(case, F.cand(cid, why)) for cid, why in candidates]
with torch.no_grad():
s = score(model, tok, F, seqs)
Read more
Details• Architecture: three models (l1_coverage_ranker, l2_context_rescorer, l1_pointwise_ablation).
• Test set: 1,160 questions (InsClosure), the ClosureRAG metric achieves FullCov@5 27.6 and R@5 69.1.
• Training: performed on a single A100-40GB GPU in bf16 format with the AdamW optimizer (lr 1e-5).
• License: Apache-2.0 (same as the base model).










Comments