serving from snapshots

this pattern solves one problem: a service that degrades when it does maintenance on itself. index rebuilds, backfills, and sync share a machine, a database file, and a page cache with live queries, so every maintenance operation is a partial outage for users.

the rule that fixes it:

no work in a request may be proportional to corpus size.

the shape

three parts, each with one job:

1. a builder runs offline — different machine, different schedule. it reads the source of truth and computes the full index, every prefix and score in sorted order, into a single artifact.
2. a manifest accompanies the artifact: checksums, schema version, row counts. the serving side validates the manifest before adopting the artifact, and the swap is atomic. the manifest is a contract, not a hint — a failed check means the previous snapshot keeps serving.
3. an overlay is a small live layer holding everything that changed after the snapshot: new records, deletions, renames, moderation actions. a query reads the snapshot, then applies the overlay to the result.

a query then reads a fixed number of precomputed rows plus a small overlay, and nothing in it grows with the corpus.

this is the lambda architecture, and its known weakness applies

the shape has a name. the builder is the batch layer, the snapshot is the serving layer, the overlay is the speed layer. jay kreps' 2014 critique names the cost: you now maintain two systems that must stay aligned, and the alignment has no natural enforcement.

the failure is concrete. a snapshot rebuild recorded as a task with no schedule left the serving snapshot 19 days stale. the overlay for one common key grew to millions of rows with no alarm. that single query took 210 seconds while every other query stayed under one second — the overlay read also had no LIMIT, so nygard's unbounded-result-set antipattern turned "large" into "210 seconds".

both halves are required to produce the outage: an invariant with no schedule, and a read with no bound.

put the batch layer where a long run costs nothing

a batch layer on infrastructure that punishes long runs will eventually attack the serving layer it exists to protect. the mechanism:

• an hourly scheduled machine with a 45-minute watchdog and a restart-on-failure policy
• build time grows with the corpus, from 17 minutes to 40
• an upstream slowdown pushes a build past the watchdog
• the watchdog kills it, the restart policy starts a fresh full-corpus scan
• the loop's primary output is load on the database the serving path reads

the correction is placement, not tuning: move the builder somewhere a batch job is allowed to be slow, behind a scheduler with a generous timeout. the failure mode of a late or failed run is then safe, because the last good snapshot keeps serving.

corollary: if the batch layer needs a strict watchdog to coexist with its infrastructure, the infrastructure is wrong for the batch layer.

this migration is invisible to the serving side by construction — it validates a manifest, not a provenance. the binary, the gates, and the artifact format do not change, so there is no downtime.

the two rules that keep it safe

• the overlay is not a cache. a cache can be dropped and rebuilt. the overlay is authoritative for every change newer than the snapshot: deletions, takedowns, renames. dropping it, or bounding it carelessly, serves incorrect data rather than stale data.
• the batch schedule carries correctness. the overlay stays small only while new snapshots keep arriving, so "the builder ran recently" is an invariant — and an invariant needs a schedule and an alarm on its age. the 210-second query was the price of three weeks without either.

related

• change-timestamps — the snapshot gate compares counts against a pinned column, which only works if that column means what the gate assumes
• watermark-semantics — the same one-cursor-one-meaning discipline for log positions
• at-least-once-forwarder
• operations/observability-of-absence — why an unscheduled rebuild produces no signal until it produces an outage

sources

• typeahead — actor search over ~7M atproto accounts; docs/retros/2026-05-25-typeahead-rearchitecture-notes.md (the rebuild), and the june 2026 overlay-bloat retro (the 19-day staleness and the 210-second query)
• pub-search — the adoption one month later, crediting typeahead: docs/snapshot-pipeline.md, docs/scaling-plan.md; docs/builder-offbox-plan.md for the july 2026 builder move (commits b535275, d3348bf)
• questioning the lambda architecture — jay kreps, 2014
• release it! — michael nygard; unbounded result sets, and much of the stability vocabulary in these notes