serving from snapshots

this pattern solves one problem. a search service became slow and incorrect when it did maintenance on itself. index rebuilds, backfills, and sync used the same machine, the same database file, and the same page cache as live queries. thus each maintenance operation degraded the service for users. typeahead, an actor-search service for atproto with approximately 7M accounts, found this limit in may 2026. the team rebuilt the service around one rule:

no work in a request should be proportional to corpus size.

the shape

the pattern has three parts. each part does one job:

1. a builder operates offline, on a different machine and a different schedule. it reads the source of truth. it computes the full index, with each prefix and each score in sorted order. it makes one artifact.
2. the builder publishes the artifact with a manifest that contains checksums, a schema version, and row counts. the serving side examines the manifest before it adopts the artifact. the swap is atomic. "the manifest is a contract, not a hint." the serving side rejects a bad build. the old snapshot continues to serve.
3. an overlay is a small live layer. it holds the changes that came after the snapshot: new accounts, deletions, renames, and moderation. a query reads the snapshot first. it then applies the overlay changes to the result.

a query now reads a fixed number of precomputed rows plus a small overlay. no part of the query does work that increases with the corpus size. pub-search, a search service for atproto publications, adopted the same pipeline one month later. its documentation credits typeahead.

this is the lambda architecture, and its known weakness applies

the shape has a name: the lambda architecture. the builder is the batch layer. the snapshot is the serving layer. the overlay is the speed layer. jay kreps wrote a critique of this architecture in 2014 ("questioning the lambda architecture"). his point: you must keep two systems aligned, and that is the weakness. typeahead paid that cost. the snapshot rebuild was a recorded task with no schedule. the serving snapshot became 19 days stale. the overlay for the key bsky grew to millions of rows without an alarm. one day, the query q=bsky took 210 seconds while all other queries stayed below one second. the overlay read also had no LIMIT clause. this is the "unbounded result set" antipattern from nygard's release it!. the missing LIMIT changed "large" into "210 seconds".

put the batch layer where a long duration has no cost

pub-search adopted the pattern but kept its builder on fly. the builder was an hourly scheduled machine with a 45-minute watchdog and an on-failure restart policy. this configuration caused alerts on three days in sequence in july 2026. build time had grown with the corpus, from 17 minutes to 40 minutes. when turso was slow, the build crossed the watchdog limit. the watchdog stopped the build. the restart policy then started a new full-corpus scan. the result was a restart loop. the primary output of the loop was load on the database that the serving path also reads. the batch layer degraded the serving layer. the pattern exists to prevent exactly this condition.

the correction was the same change that typeahead made before: move the builder to a machine where a batch job can be slow. pub-search moved the builder to the home box, behind a scheduler (prefect) with a large timeout. the failure mode of a late or failed run is safe: the last good snapshot continues to serve. the binary, the gates, and the manifest did not change. the serving side cannot know where a snapshot was built. for this reason, the migration caused no downtime. corollary: if the batch layer needs a strict watchdog to operate on its infrastructure, the infrastructure is wrong for the batch layer.

the gate pins a timestamp, so the timestamp must have a clear meaning

the builder has a count gate. the gate compares the count of copied rows with the count of rows that turso reports below the watermark. the gate has a small tolerance, for deletions that occur during the build only. the gate assumes that the set of rows below the watermark does not change. that assumption is true only if the pinned timestamp changes when the content changes. the pub-search indexer set indexed_at = now() on each write, also when the content was identical. atproto platforms frequently write full archives again. one account wrote its archive of 4,500 documents in minutes. each identical write during a build moved a copied row above the watermark. the difference between the two counts drifted by approximately 450 rows for each build. that is 8 times the tolerance. the gate could not pass while the writes continued.

the repair changed the meaning of the column, not the tolerance. indexed_at now means "the time of the last content change". the indexer keeps the old value when the content hash is equal. the assumption of the gate became true. the general rule: a snapshot gate is a contract about the meaning of the pinned column. pin a column that changes only when the data changes. if you do not, the gate measures write traffic, not correctness.

the two rules that make the pattern safe

• the overlay is not a cache. you can remove a cache and build it again. the overlay is authoritative for all changes that are newer than the snapshot: deletions, takedowns, and renames. if you remove the overlay, or limit it without care, you serve incorrect data, not slow data.
• the batch schedule carries load. the overlay stays small only if new snapshots continue to arrive. "the builder ran recently" is an invariant. an invariant needs a schedule and an alarm. the typeahead correction was a scheduled rebuild, every 3 days. the 210-second query was the cost of a recorded task that had no schedule for three weeks.

sources

• typeahead — docs/retros/2026-05-25-typeahead-rearchitecture-notes.md (the rebuild), the june overlay-bloat retro
• pub-search — docs/snapshot-pipeline.md, docs/scaling-plan.md (the adoption, with credits); docs/builder-offbox-plan.md (the july 2026 builder move and the gate/watermark incident, commits b535275 and d3348bf)
• questioning the lambda architecture — jay kreps, 2014
• release it! — michael nygard (unbounded result sets, and much of the stability vocabulary in these notes)