at-least-once delivery for an event forwarder

a forwarder sits between an upstream event source and a downstream consumer, holding a cursor against the upstream so it can resume. the naive version acks the upstream on receipt and sheds under backpressure. that combination has a silent-loss hole no amount of buffering closes: once a frame is acked, every downstream shed — queue overflow, crash, processing error — loses it permanently, and loss is indistinguishable from quiet.

three mechanisms close it, and only work together.

1. an outbox, not a ring buffer

the producer keeps every emitted frame queued until the consumer acks it. a bounded ring that overwrites is the bug; an outbox that retains is the fix.

• redeliver everything unacked on reconnect
• retransmit on a timer (riding an existing heartbeat is cheap — e.g. a 20s heartbeat with a 60s unacked resend threshold)

in steady state acks land in milliseconds and the outbox hovers at empty, so the memory cost is near zero. the outbox is only large when something is already wrong, which is the correct time to spend memory.

2. ack after durable commit

the consumer acks only once the frame's effects are committed — the database write batch, not the enqueue. a shed or errored frame is simply never acked, so it returns via retransmit.

this is the piece that changes what backpressure costs. before: backpressure discards data. after: backpressure costs latency.

3. the durable cursor is pinned to min(inflight) − 1

the producer's resume cursor against its upstream never advances past the oldest unacked or in-flight sequence. this makes outbox eviction survivable: if memory pressure forces the outbox to drop frames, a restart replays them from upstream.

it also yields the cheapest possible end-to-end check:

cursor == read position ⟺ nothing unacked

one comparison in a progress log proves the whole ack chain is flowing. no separate lag metric required.

the prerequisite

the consumer must be idempotent on the keys it touches, because every recovery path above produces duplicates. that is the trade: exactly-once delivery is not available, and at-least-once plus idempotent writes is cheap.

protocol migration

this change can be made compatible in both directions, which turns a lockstep cutover into two independent deploys:

• an old consumer that acks early causes the outbox to release immediately — i.e. the old semantics
• an old producer ignores acks entirely

designing for that is worth the effort. a delivery-semantics change that requires both sides to deploy simultaneously will be attempted during an incident, which is the worst time to discover an ordering constraint.

related

• background-tasks — the same at-least-once contract one layer up, where redis streams' PEL plays the outbox's role
• serving-from-snapshots — durability before emission, applied to read replicas
• operations/observability-of-absence — why silent loss stays open so long

sources

• pub-search's ingester→backend channel (2026-08-04), which ran the broken shape: the backend acked before processing, then drop_oldest on a bounded queue discarded frames that were already acked
• zat.dev/stream — the same protocol at full-network scale