cluster the 2d projection

mapping an embedded corpus (thousands of points) into something structure can be read off — clusters, neighborhoods, work items. the pipeline order is the part that matters, shown here from the phi-atlas flow — a daily batch that maps everything phi (a bluesky agent) knows: memories, posts, cards, goals:

1536-d vectors → umap → 2d coords → hdbscan (twice) → clusters → promotion logic

why cluster in 2d

hdbscan runs on the umap output, not the raw 1536-d vectors. density-based clustering in high dimensions is unstable (distances concentrate); umap with metric="cosine" does the neighborhood-finding, hdbscan then reads density off the 2d layout. bonus: clusters visually match the map, because they were computed on the map.

tuning that mattered:

• n_neighbors scales as sqrt(n) clamped to [5, 30] — a fixed value is wrong at both ends as the corpus grows.
• fixed random_state so consecutive daily maps are comparable.
• two hdbscan passes at different min_cluster_size (20 coarse / 5 fine), with each fine cluster linked to a parent coarse cluster by member majority — coarse for orientation, fine for work-item granularity.

the noise label is a landmine

hdbscan labels outliers -1. treat that as a cluster id and every noise point "shares a cluster" with every other noise point — any grouping logic silently degrades into one giant fake cluster. skip -1 explicitly everywhere cluster membership means anything.

structure in, work items out — deterministically

the payoff is that cluster co-membership becomes a signal no llm has to guess at. each point gets a promotion_status computed by plain rules: referenced by a public connection → connected; public layer → promoted; shares a fine cluster with a public point → promoted; with a summary → summarized; else raw. "raw" literally means "no public anchor nearby." a companion flow (the docket) then takes fine clusters with ≥3 raw points and synthesizes them into work items for the agent — promotion candidates. the whole signal — "this dense private cluster has nothing public near it" — comes from cluster membership and plain rules; no llm reads the corpus to find it.

(a lighter cousin of this: phi's live memory-graph endpoint projects per-user observation centroids with plain pca — deterministic, cheap, good enough to serve per request. umap is for the daily batch, pca for the request path.)

keep the pod alive

embedding + umap + clustering over thousands of points is memory-hot: null out intermediate vectors/content before serializing results, and collect. also, a carried scar: upload the resulting json blob to a pds as application/octet-stream — at least one pds implementation mangles blobs whose stored mime is json.

sources

• my-prefect-server flows/phi_atlas.py — the full pipeline
• my-prefect-server flows/docket.py — promotion candidates from cluster density
• bot/src/bot/memory/namespace_memory.py — the pca cousin