Skip to content

compression

nate
Aug 8, 20262 min read1 read

compression

written against zig 0.16.

std.compress.zstd (decode)

zig std can decompress zstandard — no vendored libzstd needed for plain-frame decode (this is what let zat consume jss archive blocks with zero new dependencies):

var in: std.Io.Reader = .fixed(compressed);
var d: std.compress.zstd.Decompress = .init(&in, &.{}, .{});
_ = try d.reader.streamRemaining(&allocating_writer.writer);

measure before shipping it on a hot path: on real jss archive blocks (atproto-bench jss lane, 58 frames / 61 MB decompressed) std's decoder took ~410 ms where libzstd took 22 ms — ~20× — and the gap survived every buffering strategy (allocating writer, reused window + discard, preallocated). zat ended up vendoring facebook/zstd v1.5.7 decode-only (10 C files, ~150-line build.zig addition, cross-compiles) and got 2× ahead of the go reference. std zstd is fine when the wire is the bottleneck.

sharp edges, verified in the 0.16 source:

  • decode only. there is no compressor in std.
  • verify_checksum: true panics — "not implemented yet" in Decompress.zig. frame content CRCs are silently skipped, so integrity must come from an outer layer (zat uses the jss segment xxh3 checksum; std.hash.XxHash3 has streaming init/update/final).
  • no dictionary support (DictionaryIdFlagUnsupported). fine for plain frames; not for dictionary-compressed streams like jetstream subscribe-v2 wire compression.
  • with a non-empty buffer, init asserts capacity ≥ options.window_len + zstd.block_size_max; the empty-buffer + writer form (above) is the simple path for buffered whole-frame decode.

sources

  • zat — vendored zstd v1.5.7 decode-only, jss archive block decode

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

Across the AtmosphereDiscussions