OpenToolSearch Reranker: A compact reranker for selecting AI agent tools
3 min readupdated
This model from AI3 Inc. helps AI agents efficiently handle massive libraries of functions and tools (such as function declarations or MCP tools) without overloading their system prompts. OpenToolSearch Reranker works as a second-stage search: a 0.6B-parameter cross-encoder (a modified causal LLM with a yes/no output) reads the query alongside the tool description and scores its relevance. Built on Qwen/Qwen3-Reranker-0.6B, this solution paired with a first-stage embedder outperforms many heavier alternatives while saving tokens and response time.
Key features
- Precise selection: A 0.6B cross-encoder evaluates the match between the query and tools on a binary scale (yes/no).
- Balanced architecture: Weights were averaged 50/50 with the base model, and JSON and YAML template augmentation was applied during training.
- Resource efficiency: Works alongside a fast first-stage embedder (such as
nvidia/Nemotron-3-Embed-1B-BF16) to filter out excess tools before calling the main LLM. - English optimization: Trained on cleaned ToolRet, Toucan-1.5M, and Glaive function-calling v2 datasets.
What you can do
- Organize tool search: Create a local tool index for an AI agent to pass only the necessary functions during a conversation.
- Accelerate agent pipeline: Use a two-stage search—first selecting top-50 candidates via an embedder, then reranking them with this model.
- Integrate into custom frameworks: Use the model with the
transformerslibrary to build custom API search engines.
Example
Running and evaluating tool relevance using the transformers library:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "ai3inc/opentoolsearch-rerank-0.6b"
tok = AutoTokenizer.from_pretrained(name, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16).eval()
yes, no = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids(
"no"
)
PRE = (
"<|im_start|>system\nJudge whether the Document meets the requirements"
" based on the Query and the Instruct provided. Note that the answer can"
' only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
)
SUF = "<|im_end|>\n<|im_start|>assistant\n\n\n\n\n"
DEFAULT = "Given a user request, retrieve the tools that are needed to fulfil it"
def score(request, tool_json, instruction=None):
text = f"{PRE}: {instruction or DEFAULT}\n: {request}\n: {tool_json}{SUF}"
ids = tok(text, return_tensors="pt", add_special_tokens=False).to(
model.device
)
with torch.no_grad():
logits = model(**ids).logits[0, -1]
return (logits[yes] - logits[no]).item()
tool = '{"name": "get_weather_forecast", "description": "Get the weather forecast for a city on a given date."}'
print(score("What's the weather in Seoul tomorrow?", tool))
Read more
DetailsBenchmarks: On the ToolRet benchmark (ACL 2025), pairing with Nemotron-3-Embed-1B achieves 58.37 nDCG@10 (top-100 reranking), trailing larger alternatives by only about 1.5 points while being significantly smaller.
Hardware and VRAM: 0.6B parameters in bfloat16 format take up about 1.2 GB of VRAM.
Training: 300 steps of fine-tuning on a single NVIDIA B300 GPU (learning rate 6e-6), using hard negatives from the first-stage embedder.
Limitations: Only English is supported; requires 50–100 forward passes per search query (depending on the number of candidates).










Comments