running gemma 4 12b locally via MLX

apple silicon (M5 Pro, 64GB unified) → MLX is the fastest backend.

THE KEY GOTCHA: use mlx-vlm, NOT mlx-lm

gemma 4 is a unified multimodal model — its config.json has model_type: gemma4_unified (text submodule gemma4_unified_text). mlx-lm is text-only and maps model_type → a module file; there's no gemma4_unified.py, so it dies with Model type gemma4_unified not supported.

the model lives in mlx-vlm (mlx_vlm/models/gemma4_unified/). use that. text-only prompts work fine through it — pass --image/audio for multimodal.

uv tool install "mlx-vlm @ git+https://github.com/Blaizzy/mlx-vlm"
# git main needed at release; pypi (0.6.1) lagged the gemma4_unified merge

models (mlx-community on HF)

repo: mlx-community/gemma-4-12B-it-<quant> (note the capital B).

at release several quant repos were placeholders — created but no weights pushed (only .gitattributes). check before pulling:

curl -s "https://huggingface.co/api/models/mlx-community/gemma-4-12B-it-8bit" \
  | jq '[.siblings[].rfilename] | length'   # 1 = empty placeholder

at launch: bf16 (~24GB, 12-14 files) + 6bit/5bit/4bit had real weights; 8bit/mxfp4/mxfp8 were empty (backfilled by community over a day or two).

get a real 8-bit now (self-quantize from bf16)

don't wait on the community — quantize locally with mlx-vlm's converter:

mlx_vlm.convert --hf-path mlx-community/gemma-4-12B-it-bf16 \
  -q --q-bits 8 --mlx-path ~/models/gemma-4-12B-it-8bit

downloads bf16 (cached in ~/.cache/huggingface), quantizes, writes a local mlx model dir you own. our run: 8.51 bits/weight, 12GB on disk.

downloads slow? a VPN throttles the HF/Cloudflare CDN pull hard — disable it for big fetches (took us 2 MB/s → ~19 MB/s). note: HF_HUB_ENABLE_HF_TRANSFER is now deprecated in huggingface_hub; the new knob is HF_XET_HIGH_PERFORMANCE=1 (Xet transfer). set an HF_TOKEN to dodge unauthenticated rate limits.

run

mlx_vlm.generate --model ~/models/gemma-4-12B-it-8bit --prompt "..." --max-tokens 200
mlx_vlm.chat    --model ~/models/gemma-4-12B-it-8bit          # interactive
mlx_vlm.server  --model ~/models/gemma-4-12B-it-8bit          # openai-compatible API
# multimodal: add --image foo.png (or audio) to exercise the unified backbone

--model takes a HF repo id (auto-downloads) or a local path.

measured on M5 Pro / 64GB

quant	disk	gen tok/s	peak mem
bf16	24GB	~11	24.0 GB
8bit	12GB	~21	12.8 GB

8-bit ≈ 2× faster at half the memory, negligible quality loss → the default pick. 6bit/4bit only if you want even less RAM / more speed.