nuhmanpk/toolscout-bge-small

ToolScout 🧭 (v0.1)

A tiny (33M params) embedding model that routes a user request to the best tool from any list of tools, including tools it never saw during training. Fine-tuned from BAAI/bge-small-en-v1.5 on nuhmanpk/semantic-router-dataset using each example's candidate pool as hard negatives.

Fast enough to run on CPU in milliseconds, so it can sit in front of an LLM agent and trim a large tool list down to the top few candidates before the LLM sees it.

Results

Each test query is ranked against all 36 tools in the catalog.

Model	Seen top-1	Seen MRR	Unseen top-1	Unseen top-3	Unseen MRR
BAAI/bge-small-en-v1.5 (no tuning)	98.0%	0.990	95.7%	100.0%	0.979
ToolScout v0.1	100.0%	1.000	98.6%	100.0%	0.993

• Unseen tools: 6 tools (audio_transcriber, code_debugger, creative_writer, equation_solver, file_reader, git_manager) were removed from training entirely, so this measures generalization to new tools.
• Test sizes: n=50 (seen) and n=70 (unseen). The base model is already strong on this data, so treat the gain as modest.
• Where it helps most: separating similar tools. For example, "run this and tell me why it crashes" goes to code_debugger instead of code_executor, even though code_debugger was not in training.
• The published weights were retrained on all tools after evaluation.

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nuhmanpk/toolscout-bge-small")

PREFIX = "Represent this sentence for searching relevant passages: "
tools = {
    "weather_lookup": "Gets current weather and forecasts for a city.",
    "email_sender": "Sends emails to contacts.",
    "calculator": "Performs arithmetic calculations.",
}
q = model.encode(PREFIX + "will it rain tomorrow?", normalize_embeddings=True)
t = model.encode([f"{k}: {v}" for k, v in tools.items()], normalize_embeddings=True)
print(list(tools)[(t @ q).argmax()])   # -> weather_lookup

Tips

• Always add the query prefix above.
• Format each tool as name: short description.
• For large tool lists, take the top-3 and let your LLM make the final pick.

Training

• Base model: BAAI/bge-small-en-v1.5
• Loss: MultipleNegativesRankingLoss, where each row's pool of 8 tools is the softmax (1 positive, rest as hard negatives)
• 4 epochs, lr 3e-5, warmup 10%, fp16, trained on Kaggle (T4)
• Data: 407 examples across 36 tools and 8 categories

Limitations

• Training data is small and mostly synthetic/templated; real-world phrasing may be messier.
• Single-tool selection only (no multi-step plans or multi-tool calls).
• English only.
• Works best with short, clear tool descriptions.

A v2 with harder, more natural queries, confusable tool groups and 100+ tools is planned.

Author

Built by Nuhman PK

• 🤗 Hugging Face: huggingface.co/nuhmanpk
• 💻 GitHub: github.com/nuhmanpk
• 📊 Kaggle: kaggle.com/nuhmanpk
• 💼 LinkedIn: linkedin.com/in/nuhmanpk
• 🐦 X: @pk__nuhman
• ✍️ Medium: @nuhmanpk · Dev.to: dev.to/nuhmanpk

If you find this useful, a ❤️ on the repo helps!

Citation

@misc{toolscout2026,
  author = {Nuhman PK},
  title  = {ToolScout: A Tiny Embedding Model for Tool Routing},
  year   = {2026},
  url    = {https://huggingface.co/nuhmanpk/toolscout-bge-small}
}