set input_type at both ends

embedding models with asymmetric modes (voyage's input_type) want documents embedded as "document" and queries as "query". getting this wrong doesn't error — it silently degrades recall. pub-search, a semantic-search service over atproto publications, sets it at both ends: the ingest worker embeds with input_type: "document", the query path with input_type: "query".

prepare documents deliberately

the ingest side is where quality is decided:

• truncate at ~2000 tokens. voyage-4-lite accepts 32K tokens; pub-search cuts content at 8000 chars anyway. past a couple thousand tokens the extra retrieval value per dollar is tiny, and one giant document can't eat the batch budget.
• prepend the title. title + "\n" + content — titles carry dense signal, and content is truncated from the end, so the head is the safe place for it.
• sanitize utf-8 before json. invalid byte sequences in scraped content make a json serializer emit the field as a byte array instead of a string, and the embedding request breaks in a way that looks like an api error. replace invalid sequences with spaces at the boundary.
• skip the too-short. documents under ~50 chars aren't worth a vector — they embed to near-noise and pollute neighborhoods.

validate the batch response

a batch embed response can succeed while being wrong. pub-search checks both the vector count against the batch size and each vector's dimension against the expected 1024 — a mismatch errors instead of silently mis-mapping vectors to documents, which would poison the index quietly.

batch on a poll, embed once

the embedder is a background worker polling for embedded_at IS NULL rows in batches of 20 — ingestion and embedding are decoupled, so an embedding-api outage backs up a queue instead of dropping documents. the phi-atlas flow (a daily batch that embeds an agent's whole corpus — see cluster-the-2d-projection) goes further: it reuses vectors already stored in turbopuffer instead of re-embedding (same model, same text — the vector is already paid for) and keeps a content-hash disk cache so daily rebuilds only pay for changed content.

sources

• pub-search/backend/src/ingest/embedder.zig — batching, truncation, sanitization, validation
• pub-search/backend/src/db/tpuf.zig — query-side input_type
• my-prefect-server flows/phi_atlas.py — vector reuse + content-hash cache