infra cost optimization & home-box offload

Context: I run a fleet of public atproto infrastructure out of pocket and want to (a) track what it costs, (b) bring the cost down, and (c) eventually solicit grants for the public-good pieces. This note is the durable record of what we built, what we learned, and the roadmap for moving compute onto a home Linux box. Started 2026-06; keep it current. Project names throughout are my services — most have repos under tangled.org/zzstoatzz.io or tangled.org/zat.dev.

the cost-tracking system (built, live)

A connector hub collects infra costs and writes a daily snapshot to my PDS, rendered on the hub.

• Code: my-prefect-server → packages/mps/src/mps/costs/ (connector hub: fly, cloudflare, hetzner, neon; Connector protocol; amounts in integer cents; projects.py maps service→project) + flows/costs.py (the daily flow).
• Storage: lexicon io.zzstoatzz.cost.snapshot (defined in evergreen/lexicons/), one record/day, rkey=YYYY-MM-DD (upsert). Written as zzstoatzz.io (the main identity on the self-hosted pds.zzstoatzz.io), NOT phi — creds in the operator-atproto-creds Secret block {handle, password, pds}; build AsyncClient(base_url=pds) before login or self-hosted accounts hit BadJwtSignature.
• Automation: costs Prefect deployment, daily 08:00 UTC, autonomous.
• Surfaces: hub.waow.tech (one collapsible CostsPanel, LLM + infra together, donut→bars, repo links, localStorage-sticky) reading /api/costs.json; data also queryable directly via com.atproto.repo.listRecords.
• Per-repo mandate: every cost-incurring repo has a root COSTS.md that fetches its live figure from hub.waow.tech/api/costs.json (a repo-scoped jq one-liner) rather than hardcoding — plus cleanup levers and a dated changelog.

connector accuracy notes (provenance matters)

• estimated:true flags heuristic figures vs billed. Honesty rule: a provider that can't measure RAISES/warns rather than silently reporting $0.
• hetzner: billed at list price (accurate); tokens are PROJECT-scoped (no account token) → hetzner-tokens Secret block holds a {name: token} dict, flow merges.
• fly: estimated from machine inventory (no real billing API); read-only org token.
• neon: org-scoped API key (/projects needs ?org_id); priced by plan tier (launch=$19/mo) spread across projects.
• cloudflare: account-scoped token (can't list /accounts → account id inlined, got via wrangler whoami); R2 needs Account Analytics: Read (token lacks it → omitted, not faked); billing API 403s → CLOUDFLARE_FIXED_USD=5 (Workers Paid).

THE big lesson: Hetzner US datacenters cost ~3.5× EU for identical hardware

Surfacing the data immediately exposed the dominant waste: three big Hetzner boxes were in US datacenters (Ashburn ash, Hillsboro hil), which Hetzner prices at ~3.5× the EU rate (fsn1/nbg1/hel1) for the same silicon.

Rule going forward: always provision Hetzner in EU unless US latency is genuinely required. This one move took the bill $709 → ~$309/mo (~$4,800/yr, 57%). (relay & zlay are deliberately SEPARATE EU nodes — relay 138.201.188.76, zlay 168.119.156.18; a temp compat ingress on the relay node proxies stale zlay.waow.tech traffic until caches drain — remove later.)

the home box (the next frontier)

Specs: Intel i9, 24 cores, 64 GB RAM, 1 TB SSD + 2 TB NVMe. Linux, at home. This is more compute than every non-relay cloud machine combined. So compute/RAM/disk stop being constraints — only bandwidth and uptime bind.

the principle: separate bytes from compute

A residential line is bad at exactly two things — serving sustained public bytes and ingesting the firehose — and great at everything else.

• Home box = compute & state factory. Batch/ML/index/transcode jobs that read inputs, write results to cloud (R2/Postgres), or serve a small JSON API.
• VPS (Hetzner/OVH ~$5) = public edge & bandwidth sink. Stable public IP, TLS, firehose ingest, big responses.
• Tailscale = private wire between them. Caveat: if the VPS proxies a home response, those bytes still climb the home upload — so only route LOW-bandwidth endpoints; push large artifacts to R2 and serve those from cloud. (Tailscale Funnel has the same caveat + throughput limits → prefer VPS+Caddy reverse proxy for real traffic.)
• Sharp pattern for firehose consumers: thin tap on the VPS filters the firehose to the collections you need, forwards the trickle over Tailscale to the indexer at home (holds the big on-disk index, answers queries). Bandwidth-heavy part on VPS, storage/compute-heavy part at home.

per-project verdict

phased roadmap

1. Phase 1 (zero public exposure): add a Prefect worker on the home box (process/docker work pool, polls prefect-server.waow.tech outbound over Tailscale — no ingress, no port-forward). Retarget heavy flows to it: transform (dbt), rebuild-atlas (UMAP/HDBSCAN), phi-atlas, compact (embeddings). Reversible (just change the deployment's work pool). Lowest-risk way to cash in 24 cores; they run faster than on the cramped cloud pods too.
2. Phase 2 (needs VPS edge): indexers — typeahead, pub-search, coral — via the tap-on-VPS / index-at-home split. Biggest cost wins.
3. Phase 3: local Postgres on the NVMe for dev/stg branches → trim Neon toward prod-only. Keep prod plyr on Neon for uptime.

constraints / what stays cloud regardless of how beefy the box is

Residential uptime, power, dynamic IP (hide behind VPS+Tailscale; fine for batch that catches up, bad for SLA). Funnel throughput limits. Stays cloud: relays (firehose bandwidth), the PDS (uptime + blob bandwidth + foundational), plyr user-facing.

Envelope: phases 1–2 plausibly take the bill $309 → ~$200/mo or below, with the already-sunk home box doing the work.

status: all deployments migrated (2026-06-17)

All 12 Prefect deployments run on home-pool (heavypad) with ≥1 successful run. Worker is a systemd service (my-prefect-server/deploy/home-worker/). prefect.yaml points everything at the home anchor (command + persistent analytics env).

Gotchas that cost real time (now encoded):

• stale work-queue on pool change — the (3.7.2) server keeps a deployment's old work_queue_id when you only re-deploy after changing its pool. Fix: delete + recreate the deployment (we did delete-all + deploy --all).
• uv must be on the worker's PATH — process workers spawn uv run per flow.
• persistent analytics path — process flow runs execute in an ephemeral /tmp dir, so ANALYTICS_DB_PATH/LLM_SPEND_LOG_PATH/PREFECT_LOCAL_STORAGE_PATH MUST point at a stable dir (/home/stoat/prefect-analytics) or the DuckDB pipeline doesn't coordinate across runs.
• the transform/dbt killer — transform runs fine under --python 3.13.11, but it shells out subprocess.run(["uv","run","dbt","compile",...]) with NO --python, so the nested uv venv picks 3.14 (newest matching requires-python>=3.13), where dbt-common→mashumaro + rpds-py break (rpds's compiled .so is missing for that combo). The outer --python only covers the flow process, not nested uv runs. Fix: pin the nested call (uv run --python 3.13.11 dbt ...). General lesson: any nested uv run inside a flow needs its own --python.
• DuckDB pipeline (ingest→transform→brief→compact) is an atomic unit — all must run on the same host (they share the DuckDB), so migrate them together.

Hub staleness — RESOLVED 2026-06-18, but NOT the way this note originally planned. See the session log below; the "move the hub to heavypad" plan was the wrong mechanism.

2026-06-18 session: the latency lesson (+ pagination, caching, Phase 2 start)

The hub is now fresh AND fast — and the hub did NOT move to the home box. Trying the literal plan above taught the session's main lesson:

• Joined the existing EU Hetzner node to the tailnet (hetzner-edge) — reusing the box we already pay for, no new node, $0.
• First did the planned thing: repointed hub.waow.tech → a hub container on heavypad over the tailnet. Result: ~12s per page load. Isolated it — the page renders on heavypad in 12ms; the entire ~12s was a ~230–400KB page crossing the Atlantic twice (US user → DE edge → US home) over a DERP-relayed residential link (home NAT blocked a direct path).
• LESSON (corrects the roadmap): serve bytes from the edge, sync the data — never serve a heavy page from home across the ocean. The data is tiny (~4MB: hub.duckdb + llm-spend.jsonl); the page is not. So: the EU k8s hub serves the page (EU-local, ~1–1.6s), and a 3-min cron on heavypad rsyncs the two data files to the edge's hostPath over the tailnet (my-prefect-server/deploy/hub-data-sync/). Fresh (≤3 min lag) AND fast. This is literally the "separate bytes from compute" principle at the top of this note: the hub is bytes, so it belongs at the edge.

Pagination. The landing page rendered every card into both a desktop table and a mobile list (~400KB SSR doc), blocking hydration (the cost panel couldn't open until the whole doc downloaded). CardTable now renders the top 40 sorted rows and reveals the rest on demand — caps the SSR render as the corpus grows.

Prompt-caching cost story (full writeup: my-prefect-server/docs/prompt-caching.md). Surfaced provider/model + the realized cache-hit rate on the hub. The aggregate ~11–13% is correct-low: the minimum cacheable prefix applies to the cached prefix (system prompt), not the whole request — haiku's floor is 4096 tokens, so small-prompt haiku flows (compact/brief/phi_atlas) cache nothing despite the setting; morning (sonnet) clears the floor but isn't configured to cache; only curate/docket/phi (cleared floor + configured) actually cache. compact's cost is call-count driven (~10× any other flow), which caching can't touch.

typeahead-index — Phase 2, FIRST INDEXER CUT OVER (2026-06-18). The Zig snapshot indexer is lifted off Fly onto home-pool (flows/typeahead_index.py) and is now live: prod channel, schedule 0 9 */3 * * active. Validated in two stages — a local-channel run (e3b70e03) proved build→index, then a staging run published a real 10GB snapshot to R2 (staging/heavypad-validation/…). First prod build fires 2026-06-19 09:00 UTC; the Fly indexer stays as a backup publisher until ≥2 clean heavypad prod builds (watch latest.json builder_version flip dev→heavypad), then it's retired (−~$16/mo). Measured facts that shaped it:

• uplink is ~28 Mbit/s → a 10GB upload saturates it ~45 min; RCLONE_BWLIMIT=10M is a no-op (above the line). Cadence dropped to every 3 days (~130GB/mo) and runs overnight (09:00 UTC).
• each prod run ≈ 75 min (≈30 build + ≈45 upload). Build dir self-cleans.
• gotchas (in repo memory): typeahead has its OWN turso blocks (typeahead-turso-*; the shared ones are the leaflet db); INDEX_SAMPLE_LIMIT=0 required (default 1000); rclone path resolved via INDEX_RCLONE_BIN (binary no longer hardcodes /usr/local/bin); prefect 3.7.2 won't overwrite an existing schedule → delete+recreate the deployment to change it. The indexer clones typeahead's remote at runtime, so local commits must be pushed or it builds stale.

Process lesson. Leaving my-prefect-server parked on a feature branch let another agent build its work on top of it. Keep the main checkout on main between tasks — several agents touch these repos concurrently.

still open (deferred, not lost)

• typeahead-index: confirm the first prod build (2026-06-19) flips latest.json builder_version→heavypad, then retire the Fly indexer machine + 100GB volume and update typeahead COSTS.md (−~$16/mo). Optional follow-up: lower RCLONE_BWLIMIT below ~3.5 MB/s if the ~45-min uplink saturation bothers; Prefect alerting on the run.
• relay EU-migration infra-as-code uncommitted on its ops/... branch.
• the tailscale-edge scaffold has a local git init but no remote yet.

related

• the generalizable idea: cost-declaration — a standard atproto protocol for declaring costs (this whole system is a v0 reference impl).