harness weight destabilizes weak models

the agent harness around a local model has a measurable cost: a heavy harness (big system prompt, a dozen built-in tools) breaks tool-calling that the same model performs cleanly under a lean one. observed with Pi (@earendil-works/pi-coding-agent), a lean headless coding agent that points at any OpenAI-compatible endpoint.

lean beats heavy (for weak models)

the load-bearing finding: a heavy harness (big system prompt + a dozen built-in tools) destabilizes a weak model's tool-call formatting — it starts fencing/narrating calls that it emits cleanly under a lean prompt. opencode broke gemma this way; Pi (minimal read/bash/edit/write + a terse prompt) let the same model tool-call fine. when a local agent flails, strip the harness before blaming the model.

headless + parseable

pi -p --mode json --no-session --provider local --model "$MID" "<task>"

• -p non-interactive, --mode json emits an event stream.
• the clean place to extract the tool-call trace: message_end events with message.role == "assistant" carry the completed toolCall items (id / name / arguments). dedupe by id; partial toolcall_* deltas are noise.
• native agentskills.io skills via --skill <dir> (no bespoke skill-injection); built-in tools mean the model writes files and runs bash (e.g. zig test) itself.

pointing Pi at a local model

~/.pi/agent/models.json — add a provider for the local OpenAI-compatible server. compat.supportsDeveloperRole:false matters for OpenAI-compat servers (mlx, vLLM, ollama) that don't understand the developer role:

{
  "providers": {
    "local": {
      "baseUrl": "http://localhost:1234/v1",
      "api": "openai-completions",
      "apiKey": "not-needed",
      "compat": { "supportsDeveloperRole": false, "supportsReasoningEffort": false },
      "models": [ { "id": "/Users/me/models/gemma-4-12B-it-8bit" } ]
    }
  }
}

then serve the model (mlx_vlm.server --model ... --port 1234, see tool-calling) and run Pi against --provider local.

as an eval runner

this stack makes a clean local eval loop: one stdlib script orchestrates pi (the agent) and a compiler/tests (the objective oracle) as subprocesses — no agent framework of your own. ablate a behavior by toggling --skill; score with the compiler, not an LLM judge. parse the JSON events for what the agent actually did. (built this for zigman.)

aside: the wrong npm package bites — @mariozechner/pi is a vLLM-pod manager (bin pi-pods); the coding agent is @earendil-works/pi-coding-agent (bin pi).