fibers

written against zig 0.16.0 (LLVM < 23).

stackful fibers in zig mean a hand-written context switch in inline asm. that puts you in a place where the compiler's assumptions and your assumptions have to agree exactly, and where they disagree the failure is a segfault under optimization and nothing at all under Debug.

the failure signature to recognize

Debug        16/16 pass
ReleaseSafe  13/16 pass, 3 crash -- SIGSEGV at address 0x1

Debug clean + release crashing + faults at garbage or near-null addresses is almost always the compiler keeping a live value in a register that your asm overwrites. it is not a logic bug; do not go looking for one first. debug builds never keep values in registers across statements, which is why the crashes are ReleaseSafe/ReleaseFast-only, and why they appear/disappear across compiler versions and platforms — register-allocation luck, not fixes.

root cause: clobbers get silently ignored

std/Io/fiber.zig (byte-identical 0.16.0 ↔ master 0.17-dev.1282) switches stacks with inline asm that saves only sp/fp/pc and declares every other register clobbered, so LLVM must spill anything live across the switch. the model is sound — if the clobbers are honored. two are not:

1. ~{x30} is silently dropped by LLVM < 23 (llvm/llvm-project #167783): aarch64 clobbers of x29/x30 written by xN name are ignored; only the ABI names fp/lr register. clang always emits ~{lr} so C never hit this. zig lowers .x30 = true to ~{x30} verbatim → LLVM happily keeps live values in the link register across the switch. the fiber body executes bl instructions, so on resume the caller reads back a return address — which is why crash addresses in this class look like instruction bytes.
2. x18 is missing from the clobber list entirely. x18 is the reserved platform register on darwin/windows (never allocated, omission harmless) but a general-purpose allocatable register on linux — where LLVM parks values in it across the switch. platform-dependent corruption from a platform-independent-looking list.

the fixes:

// in the aarch64 clobber list:
.x30 = true   →   .lr = true    // spelling LLVM honors on every version
              +   .x18 = true   // required on linux, harmless on darwin

third, for anyone building their own fibers: x86_64 SysV expects rsp ≡ 8 (mod 16) at function entry (as-if a call pushed the return address). entering a fresh fiber stack via jmp with a 16-aligned rsp GP-faults on the first aligned SSE spill. bias the initial rsp by 8.

with the two clobber fixes, a stress gauntlet (100k switches, 256 interleaved fibers, cross-thread migration) passes on macos-aarch64, linux-aarch64, and linux-x86_64 — Debug and ReleaseSafe. stackful fibers are fully compatible with ReleaseSafe; the folklore to the contrary was these two clobbers. this is the crash class behind Io.Evented's ReleaseSafe crashes, upstream's #35712.

the proof shape (worth remembering as a technique)

• -femit-llvm-ir showed ~{x30} present in the IR at the exact asm site
• objdump -d showed mov x30, x1 before the switch and add x8, x30, #1 after the resume label — LLVM consumed a value through a declared clobber
• IR-says-clobbered vs disasm-says-live-across is a complete, arguable proof of a backend bug; no debugger needed

broader lessons:

• inline-asm clobber lists are trusted, not verified. nothing checks that the backend honors them (zig #18292 is the adjacent invalid-string case). a clobber that compiles is not a clobber that took effect: when a crash smells like register corruption across asm, diff IR against disasm rather than trusting the source.
• register names are not portable spellings. x30/lr are the same register with different bug surfaces; a clobber list correct on one OS (x18 on darwin) can be incomplete on another.
• "works in this build" is weak evidence for fiber code. allocation luck masks these bugs; a targeted stress harness (many switches, register pressure inside fiber frames) exposes them in a handful of tests.

test on the arch you deploy to, natively

"x86_64 covered under emulation" is not coverage: the ReleaseSafe crashes above showed up the first time the suite ran on real x86_64 hardware. for a project whose deploy target is x86_64-linux-gnu, that gap is the whole ballgame. a throwaway fly machine costs a couple of minutes:

fly apps create <name> --org personal
fly machine run ubuntu:24.04 --app <name> --region ord \
  --vm-cpu-kind shared --vm-cpus 8 --vm-memory 8192 --restart no -- sleep 86400
fly ssh console -a <name> -C "sh -c '<cmd>'"   # note: -C needs the sh -c wrapper
fly apps destroy <name> --yes

two snags worth knowing: fly ssh console -C does not parse shell metacharacters, so wrap everything in sh -c '...'; and a tangled archive URL built from a branch name containing / returns HTML, not a tarball — use the commit sha.

fiber stacks: malloc vs mmap

a fiber stack from allocator.alignedAlloc costs its full size up front and an overflow silently walks into whatever the allocator put next to it. an anonymous mapping is populated lazily — the fiber costs the pages it touches — and a PROT_NONE page below it converts overflow into a fault at the point of overflow.

at the scale fibers exist for this is not a micro-optimization. 2,800 fibers at a 256 KiB stack is ~700 MiB committed vs. tens of MiB touched.

const base = try std.posix.mmap(null, usable + page,
    .{ .READ = true, .WRITE = true },
    .{ .TYPE = .PRIVATE, .ANONYMOUS = true }, -1, 0);
// stacks grow down, so the guard goes at the low end
if (std.c.mprotect(@ptrCast(base.ptr), page, .{}) != 0) return error.OutOfMemory;
const stack = base[page..];

gotcha: with a guard page, a fiber test suite can hang on macos and pass on linux — DebugAllocator's captureStackTrace walking past the fiber stack base is harmless against a malloc'd stack, a fault against a guarded one, after which the segfault handler re-faults on the same fiber stack. it is a testing-allocator interaction, not a production path (release builds using c_allocator capture no traces), but it makes zig build test on a mac useless for guarded fibers.

also: derive the stack pointer from stack.len, not from the size constant you asked for. mmap rounds up to a page and the two stop being equal.

don't put blocking work on fibers

worth stating because the memory numbers tempt you. a fiber that makes a blocking call occupies its loop thread for the duration. measured with a Threaded postgres pool driven from fibers, 8-connection pool, 256 callers, 20ms queries:

the 0.02ms is the tell: fibers never contend for the pool, because only one of them runs at a time. correctness was perfect in both columns (20,480 ops, zero wrong results) — this is purely a throughput property.

so the two-io split is the design, not a workaround: fibers for sockets, a plain Io.Threaded for database pools, disk, and worker pools.

corollaries, found the hard way in a relay under load:

• shared mutexes/condvars must live on the fiber-aware io. a fiber that parks through a Threaded io's futex blocks its loop thread (contended lock = kernel futex wait; a backpressure poll = thread sleep). a dual-domain futex vtable (fibers to the sched table, foreign threads to the kernel word, wakes hitting both) makes one Io.Mutex safe from either side.
• never detach work that borrows a fiber frame. handing a worker thread a queue living on the calling fiber's stack and returning immediately means slow completions finish after the frame is gone and scribble recycled fiber stacks. symptom in the field: a callee-saved register's low 16 bits (an error code) arriving at @errorName as garbage — a data fault in the error-name table read, minutes into a ~1,700-fiber run. the fix is to park the fiber on a completion word until the spilled work finishes: the frame outlives the writer and the loop thread never blocks. and the spill must be a plain detached std.Thread — Threaded futures must be awaited to be freed, and Threaded's await/futex bookkeeping is undefined on threads it does not manage.
• a fiber read path that parks on EAGAIN requires a non-blocking fd — and fds created by a Threaded path (HostName.connect delegates lookup+connect inward) arrive blocking. the failure is vicious: busy sockets almost always have data, so reads return and everything looks fine, until one quiet socket blocks read(2) and seizes the whole loop. flip the fd non-blocking at the top of the fiber path, like accept does for its listener. diagnostic tell: every thread of the process in futex_wait/nanosleep wchan, none in epoll_wait or recv.

the readiness poller needs per-direction arms

epoll holds ONE registration per fd. a one-shot arm implemented as EPOLL_CTL_MOD with a fiber-pointer token means the second fiber to arm an fd — a writer arming while a reader is parked — silently replaces both the interest and the token: the reader's wake is lost or delivered to the wrong fiber. under connection churn fd numbers recycle constantly, so this fires in production shapes and almost never in clean benchmarks. the test that caught it parks a reader and a writer on one socketpair fd and expects both to wake — it passes in isolation and fails inside the full suite, which is itself the lesson: readiness bugs need dirty fd-number environments.

fix shape: a per-fd table of independent read/write tokens, merged kernel interest, per-direction dispatch, one-shot by narrowing interest after delivery. kqueue never had the problem — EVFILT_READ/WRITE are separate registrations with separate udata.

keep racing concurrency out of hot paths that don't need it

HostName.connect races every resolved address through an Io.Group and cancels the losers — the right default for a user-facing client, and a liability for a server reconnecting to thousands of hosts: per-connect it spawns a group, N member tasks, and a cancel storm, multiplying the scheduler interleaving surface by orders of magnitude. resolving and then trying addresses sequentially on the one fiber that wants the connection removes an entire class of corruption triggers from the hot path — and can connect a ~1,900-host fleet faster than the racing version, because the loop thread stops drowning in group bookkeeping.

debugging fiber-runtime corruption: instrument the park, not the crash

a corrupted callee-saved register surfaces wherever the value finally misbehaves — arbitrarily far from the write. the escalation ladder that worked: (1) checksum the switch-frame at park, verify at resume — catches scribbles in a 56-byte window and names the fiber; (2) exact intrusive-list membership booleans + a saved-stack-pointer-in-own-stack assert — catches structural aliasing; (3) snapshot 4KB of parked stack, diff at resume and on periodic sweeps, report offset + old/new bytes + the park site's return address, then continue — because some parked-frame writes are legal (workers filling stack-resident request structs), and killing the run on the first one hides the fatal one. pair with a crash farm: auto-restarting runs that save a bounded log tail and one core per death, so evidence accumulates unattended instead of one crash per babysat session.

keep the architecture switch in one helper (Context names the field rsp on x86_64 and sp on aarch64/riscv64); diagnostics are cross-platform code too, even when the snapshot they guard is comptime-gated to one architecture.

a sleep that does not park is a busy-wait that owns your loop

a fiber scheduler that keeps time in milliseconds and converts Io.Timeout durations with @divTrunc turns io.sleep(100us) into a deadline of now: the while (nowMs() < deadline) check is false immediately and the call returns without ever parking.

the caller shape that turns that into an outage is completely ordinary:

while (!done) io.sleep(Io.Duration.fromMicroseconds(100), .awake) catch {};

under Io.Threaded the spin is harmless — one of thousands of OS threads, preempted by the kernel. under a single-threaded fiber runtime, one fiber that never parks owns the scheduler. everything else on that thread starves: accept loops go deaf, io throughput collapses, and the process still looks alive because fibers already in the run cycle keep making progress. it scales with whatever drives the polling, so larger deployments fail sooner — a smaller soak only shows latency spikes that are easy to dismiss.

round durations UP. a non-zero sleep must be at least one tick of your clock or it is not a sleep.

and note where the evidence had to come from: a starved loop cannot answer HTTP, so metrics-over-HTTP could not observe its own failure. loop-internal counters printed to stderr showed 1,772 polls/sec returning 238 events/sec — a loop spinning on nothing — which named the bug in one reading.

see also

• ../concurrency.md — threads, mutexes, atomics
• patterns.md — backend selection, Evented production experience

sources

• zio — src/fiber/, docs/fiber-gauntlet-2026-07-09.md; clobber bugs root-caused 2026-07-09
• zlay (AT Protocol relay) — production evidence for the blocking-work, poller, connect-racing, and sleep-truncation sections, 2026