tool-calling on local models

the thing that actually breaks when you point an agent at a local model is whether the serving layer turns the model's output back into structured tool_calls. two days of failures here, all per-engine and per-model, and every one traced to a chat-template or parser mismatch rather than model capability.

run does-it-tool first

before building anything on a local model, check it can tool-call at all with does-it-tool — a 6-case pydantic-ai probe (single / structured-args / abstain / parallel / numeric-fidelity / dependent-chain) over any OpenAI-compatible endpoint:

uv run pytest --model <name> --base-url http://localhost:PORT/v1 -q

if cases fail with the reply leaking raw JSON into the text ({"name": "...", ...}), the model wanted to call the tool but the server didn't parse it — a template/parser problem, not the model.

the per-engine quirks I hit (ollama)

• gemma3-tools — gemma3 has no native tool-calling, so the Modelfile coerces a bare JSON array and ollama string-parses it. works for simple calls (does-it-tool 6/6) but fences code-bearing calls in ```json, which the parser drops.
• qwen2.5-coder:7b (stock) — ollama's template tells it to emit <tool_call>…</tool_call>, but the model emits bare JSON without the tags → unparsed. 5/6 does-it-tool cases fail.
• llama3.1:8b — native tool template, fine on trivial args, but narrates code-bearing tool calls as prose instead of emitting them.

three models, three different serving-layer mismatches. hand-tuning a Modelfile per model is a rabbit hole.

what fixed it: gemma 4 via mlx_vlm

mlx_vlm.server serving gemma-4-12B returns proper structured tool_calls (a separate response field, not text) — so the call can't be fenced or narrated away:

curl -s localhost:1234/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "...gemma-4-12B...", "messages":[{"role":"user","content":"write x.txt = hi, use the tool"}],
  "tools":[{"type":"function","function":{"name":"write_file","parameters":{...}}}]}'
# -> choices[0].message.tool_calls = [{function:{name:"write_file",arguments:"{...}"}}]

(serving gemma 4 itself: use mlx-vlm, not mlx-lm — see gemma-4-12b-mlx.)

the rule

when the serving layer fights you, swap the engine or move up a model generation — don't hand-tune Modelfiles. structured tool-calls beat any text-parsed format because they're immune to the model fencing or narrating the call away.

caught me when wiring a local agent for the zigman eval: gemma3/qwen2.5 on ollama never reliably tool-called; gemma-4 on mlx_vlm did, first try.