Skip to content

synthesize retrieved memories before injecting them

nate
Jul 9, 2026 · 2 min read · 1 read

synthesize retrieved memories before injecting them

phi (a bluesky agent with turbopuffer-backed memory) used to take the top 10 vector hits for the current conversation and paste them into its prompt as a block. the concrete failure: a months-old memory saying "waiting on the relay fix" ranked right next to a fresh one saying the fix shipped — cosine similarity scores both as relevant, and the model read both as current.

the fix, in get_episodic_context: the top 10 go to a small cheap model (haiku) along with the agent's goals and the current query. it dedupes near-identical hits, keeps the newer one when two conflict, marks entries that look outdated, and returns an empty string when nothing is actually relevant — so an irrelevant query injects nothing instead of ten weak matches.

phi's per-user observation blocks skip this synthesis step, because that store is cleaned at write time instead: new facts go through an extraction → reconciliation pipeline that supersedes stale rows before they're ever stored. clean on write or clean on read — one of the two has to happen before the prompt.

supersession mechanics

the write-side cleaning never deletes. updating an observation means patch_rows sets the old row's status to superseded and a new row is written with a supersedes back-link to it. every read filters status != superseded. history stays queryable.

two deliberate asymmetries in the reconciler:

  • UPDATE unions the old row's source_uris into the new row (a refinement keeps its evidence trail). DELETE-and-replace starts the new row's source_uris fresh (a correction shouldn't inherit the evidence of the claim it's overturning).
  • injected text carries explicit trust labels: llm-written summaries get "trust: low, may contain hallucinations", extracted observations get "trust: medium" plus a citation tail like (3 sources, 2w ago), verbatim interaction logs get "trust: high".

pick top_k per call site

the same store serves different jobs with different k in phi:

jobtop_k
find the one observation a new fact might supersede3
context candidates for the synthesis pass10
"have we ever talked to this person"2

sources

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

Across the AtmosphereDiscussions