Tool Concierge
MCP became a cross-vendor standard almost overnight, and a model that hallucinates a tool name or malforms an argument turns that integration into a silent failure. I fine-tuned Qwen3-8B-Base with LoRA to generate one correct MCP tool call — or explicitly abstain — instead.
Narrowing the scope on purpose
Prior work like Gorilla targets ML model-hub APIs and TinyAgent targets generic JSON tool calls. I narrowed scope instead to the MCP schema specifically, and treated abstention as a trained output rather than a fallback bolted on afterward. Training data came from 9,300 sampled rows of Salesforce's xlam-function-calling-60k, reformatted into MCP-style inputSchema JSON, plus a self-generated set of "no applicable tool" negatives.
Call accuracy went from 523/929 to 698/929.Abstention accuracy more than doubled, from 9/31 to 22/31.
LoRA on a single A100
I applied LoRA adapters to q_proj and v_proj and swept three rank/alpha/dropout combinations for one epoch each. The high-capacity config (r=64, alpha=64) won on eval loss and became the adapter used for every reported result. Every prompt, training and eval alike, runs through a fixed five-step scaffold: restate the goal, check each tool's schema, pick one tool or none, fill in arguments, then emit exactly one JSON object.
What the benchmarks actually showed
Across BFCL, Nexus, and my own held-out MCP test split, fine-tuning produced a large abstention gain and a real call-accuracy gain — but MMLU accuracy dropped slightly (24.2% → 21.4%), both numbers close enough to chance that the signal is weak, though directionally consistent with LoRA's known catastrophic-forgetting risk. BFCL's false-call rate also rose even as abstention accuracy improved, which on inspection was mostly an output-format fix (pretrain often couldn't produce a parseable answer at all on irrelevance examples) rather than genuinely better judgment. I wrote those caveats into the model card rather than let a clean headline number hide them.