cbx-rag-context-utilization-68M
fast RAG context utilization scorer
2 min readupdated
A compact model from CobuildX-AI that checks which exact sentences from the retrieved context an AI model actually used to generate its answer. You pass in a question, text passages, and the generated answer, and it outputs a score and flag for each sentence along with the overall utilization fraction (following the RAGBench standard). Built on the jhu-clsp/ettin-encoder-68m encoder (68M parameters), it belongs to a four-model lineup of specialized RAG (retrieval-augmented generation) evaluators. Unlike heavy LLM judges such as GPT-3.5, it runs locally in fractions of a second on a GPU and achieves a lower RMSE error on benchmarks. It is built for RAG developers looking to automate retrieval evaluation, debug chunking strategies, and filter out noisy context.
Key features
- Ultra-lightweight architecture: only 68M parameters, running a single pass over several short passages in 0.12s on an NVIDIA TITAN RTX (fp16) or 3.04s on a CPU.
- Sentence-level granularity: scores every context sentence individually, applies isotonic calibration to the probabilities, and computes the overall utilization fraction.
- Adaptive profiles: supports a
contractmode for long legal agreements and afinancialmode for shorter financial reports packed with numbers and tables. - Smart prefilter: in the
contractprofile, selects the top 64 BM25-matched sentences and their neighbors, retaining 98.8% of actually used sentences while scanning only 39% of the text.
What you can do
- Replace expensive LLM-as-a-Judge calls: plug it into your RAG evaluation CI/CD pipeline instead of commercial model APIs to compute Context Utilization on English documents.
- Tune your retriever and chunk size: spot dead-weight chunks that consistently make it into the prompt but get ignored by the answer generator.
- Audit legal and financial knowledge bases: verify assistant answers over contracts and financial filings using the built-in domain profiles.
Example
from cbx_rag_eval import ContextUtilization
judge = ContextUtilization(".")
out = judge(question, passages, answer)
print(out["utilization"])
used = [s for s in out["sentences"] if s["utilized"]]
Read more
DetailsBenchmarks (answer-level RMSE, lower is better): on CUAD contracts — 0.039 vs. 0.059 for GPT-3.5; on FinQA reports — 0.033 vs. 0.134 for GPT-3.5.
Sentence-level performance: AUROC 0.981 on CUAD (precision 0.78 / recall 0.60) and 0.995 on FinQA (precision 0.90 / recall 0.88).
Training and limitations: fine-tuned on RAGBench data (CUAD and FinQA) with LLM-generated labels; English only, Apache-2.0 license.










Comments