Skip to content

within my ken

fuzzy find my breadcrumbs

nate
Apr 10, 20263 min read3 reads
1

here's a new member of the waow.tech family!

ken

fuzzy find any record in your atproto repo.

"ken" isn't only Barbie's man, its Scots for "know"

ok but what does this do

type in a query, get your "related" atproto records back!

you can share a read-only view of results for a given query!*1

methodology

there's only 1 vendor here, fly.io for compute. how does it work?

you don't have to save the index if you don't want to. i just do because if i show up again, having creating new records, i don't want to wait for it to stream my CAR and re-embed everything. ken will incrementally index what is new if you save the index!

limitations and expansions

  • only text embeddings (for now! https://find-bufo.com/)*2
  • i limit how many records i index from realllly big repos
  • i should probably turn the current walk-then-materialize-then-embed pipeline into a streaming walk-to-embed pipeline to minimize peak resource utilization
  • i am not yet sure on the best architecture for this app in general, i just thought storing the pack on my PDS so i didn't have to re-embed was kinda fun

i will likely find a way to give this to @phi.zzstoatzz.io so it can find its own records on demand (it uses records on its PDS often for "global state things") but beyond that i have no immediate plans.

here's the code:

zzstoatzz.io/ken

fuzzy find my records

have a great weekend :)

P.S.

this is partially inspired by talking to @offline.arushibandi.com and @eagraf.dev about @habitat.network


  1. though of course, if you save the pack to your PDS, anyone can go grab it... but you don't have to save it to your PDS
  2. you can imagine all sorts of cool stuff here. embedding/search over images is very well-precedented, i embed/search over audio in plyr.fm, i know people can do the same for video

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

Across the AtmosphereDiscussions