finding the bottleneck

a pipeline running at a tenth of its transport ceiling and a two-hundredth of its storage ceiling is not short on resources. it has one serial stage doing accidental work. the useful skill is a diagnostic order that finds that stage without a rewrite.

the ladder, cheapest first

1. bound the system from outside. before profiling anything, establish what the ceiling actually is:

• a bare client against the same upstream, on the same box, answers "is the transport the limit?"
• iostat answers "is the disk the limit?"
• per-thread CPU (ps -T --sort=-pcpu, or deltas from /proc/<pid>/task/*/stat) answers "who is actually working?"

one thread pinned while everything else is idle is a serial core. that finding ends the conversation about worker counts.

2. profile the real workload, not a benchmark. perf record -g -p <pid> for twenty seconds on production traffic names the top symbol with a call stack. benchmarks reproduce the code path you already suspect; the real workload reproduces the one you don't.

the recurring shapes it surfaces are accidental O(n) or accidental remote work on the hot path, and they are rarely the suspected component:

• a remote call per item. a cache miss that resolves through an external HTTP dependency puts a third party's rate limit in your inner loop.
• expensive work on the dispatch thread. a worker pool is only parallelism if the expensive work happens inside it. a dispatcher that parses before handing off has a pool of idle threads and a single-threaded parser.
• observability on the observed path. publishing a gauge per item, where publishing iterates a large map, makes the measurement the bottleneck.

3. read the reference implementation's source, not its reputation. when porting or mirroring a design, the pinned upstream is already in the module cache. reading it settles design questions that comments confidently get wrong, and it is minutes of work against hours of inference.

the sequence is the point

each fix promotes the next stage to bottleneck, so the throughput curve is a staircase, not a slope:

remote call per item      1.5/s
serial parse              360/s
O(n) bookkeeping per item 768/s
transport                 3016/s   (bare-client ceiling ~3700/s)

you are finished when the limit is a resource you chose rather than code you wrote by accident. the last number's proximity to the externally measured ceiling is the proof — without step 1 there is no way to know whether 3016/s is good.

measurement discipline

• one variable per deploy. two changes in one deploy make the staircase unreadable and hide a regression behind an improvement.
• a monotonic counter over a stated window. instantaneous rates during warm-up produce confident wrong slopes; a burst on reconnect looks like a fix.
• distrust a prior conclusion that scaled the wrong knob. "4x workers bought 1.23x" is not evidence that the write path is slow. it is evidence that the workers were empty.

related

• bounded-scans — when the accidental work is I/O rather than CPU
• instance-sizing — the same measure-first discipline applied to what you rent

sources

• stream — docs/ingest-speedup-2026-08-06.md; live ingest from 1.5 to 3,016 events/s in three single-variable deploys