local models

running local LLMs on apple silicon for real agentic work — serving, tool-calling, and the harness around them.

the recurring finding across these notes: every failure in this setup traced to the serving layer (tool-call parsing, chat templates) or to the model generation — so when something breaks, swap the engine or move up a model generation before hand-tuning anything. the combination that worked here was the gemma 4 family on an engine that returns structured tool calls.

notes

• gemma-4-12b-mlx — serving gemma 4 on apple silicon; use mlx-vlm, not mlx-lm
• tool-calling — the real wall for local agents: per-engine tool-call quirks, why does-it-tool is the diagnostic to run first, and the engine that finally returned structured tool_calls
• harness-weight — a heavy harness breaks tool-calling that the same model handles cleanly under a lean one (observed with Pi)

stack that worked

Pi (@earendil-works/pi-coding-agent) → gemma-4-12B served by mlx_vlm.server on :1234 → the zig test compiler as an objective grader. all local, no GUI, no cloud keys. (LM Studio drops in on the same port if you want its GUI introspection; for a scripted loop, MLX is the better fit — headless and start/stoppable.)

sources

• does-it-tool — pre-flight: does this model actually tool-call on this endpoint?
• zigman — the eval loop these notes came from