cluster the 2d projection
cluster the 2d projection
apping an embedded corpus (thousands of points) into something structure can be read off — clusters, neighborhoods, work items. the pipeline order is the part that matters, shown here from the phi-atlas flow — a daily batch that maps everything phi (a bluesky agent) knows: memories, posts, cards, goals:
1536-d vectors → umap → 2d coords → hdbscan (twice) → clusters → promotion logic
why cluster in 2d
hdbscan runs on the umap output, not the raw 1536-d vectors. density-based
clustering in high dimensions is unstable (distances concentrate); umap with
metric="cosine" does the neighborhood-finding, hdbscan then reads density off
the 2d layout. bonus: clusters visually match the map, because they were
computed on the map.
tuning that mattered:
n_neighborsscales assqrt(n)clamped to [5, 30] — a fixed value is wrong at both ends as the corpus grows.- fixed
random_stateso consecutive daily maps are comparable. - two hdbscan passes at different
min_cluster_size(20 coarse / 5 fine), with each fine cluster linked to a parent coarse cluster by member majority — coarse for orientation, fine for work-item granularity.
the noise label is a landmine
hdbscan labels outliers -1. treat that as a cluster id and every noise point
"shares a cluster" with every other noise point — any grouping logic silently
degrades into one giant fake cluster. skip -1 explicitly everywhere cluster
membership means anything.
structure in, work items out — deterministically
the payoff is that cluster co-membership becomes a signal no llm has to guess
at. each point gets a promotion_status computed by plain rules: referenced by
a public connection → connected; public layer → promoted; shares a fine
cluster with a public point → promoted; with a summary → summarized; else raw.
"raw" literally means "no public anchor nearby." a companion flow (the docket)
then takes fine clusters with ≥3 raw points and synthesizes them into work
items for the agent — promotion candidates. the whole
signal — "this dense private cluster has nothing public near it" — comes from
cluster membership and plain rules; no llm reads the corpus to find it.
(a lighter cousin of this: phi's live memory-graph endpoint projects per-user observation centroids with plain pca — deterministic, cheap, good enough to serve per request. umap is for the daily batch, pca for the request path.)
keep the pod alive
embedding + umap + clustering over thousands of points is memory-hot: null out
intermediate vectors/content before serializing results, and collect. also, a
carried scar: upload the resulting json blob to a pds as
application/octet-stream — at least one pds implementation mangles blobs
whose stored mime is json.
sources
- my-prefect-server
flows/phi_atlas.py— the full pipeline - my-prefect-server
flows/docket.py— promotion candidates from cluster density - bot/src/bot/memory/namespace_memory.py — the pca cousin
Did you enjoy this article?
Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.