habitatsapatprotopermissioned-dataspaces

Habitat's road to release: 02 sap for syncing permissioned spaces

A package to help apps keep permissioned data up-to-date.

Aug 14, 20266 min read3 reads

ast week, we released our Space Proxy, which makes it possible for any atproto dev to experiment with the upcoming simplespace API today, ahead of the official release!

This week, we are making available sap, a Golang package for syncing permissioned spaces, inspired by the tap tool for backfilling and live-tailing public records.

How sap works:

sap implements the sync protocol from the permissioned data proposal. It crawls permissioned spaces a user has access to, pulls the records from these, and receives notifications of new writes.
Apps can use sap to continuously poll for new records in a space, and then index these into their AppView. Today, this indexes just the records and not the permission, but we have some updates that will improve the dev-ex here soon (see below).
Sap uses listRepoOps to incrementally update its synced repos and maintains a set hash for each. If a set hash deviates from the response's commit info, sap will mark the repo as desynced and do full state recovery via getRepo.
Sap also tracks server errors and will retry requests with backoff.
Currently sap is compatible with Habitat's implementation of the permissioned data proposal which use the network.habitat.* namespace and diverges slightly from the spec. We will continue to align these APIs as the finalized spec is released.

Follow ups:

Support for running sap as a stand-alone binary so that you can include it as a service for apps written in languages other than Golang.
Sap currently doesn't tell the caller about who has permission to what. At request time, your app will need to attempt to get a space credential on behalf of the requesting user to enforce permissions properly. We are working to build this into sap directly, to reduce additional steps done by the app.
Sap crawls a user's spaces via listSpaces when the user session is added via Sap.AddSession. However, the spaces protocol doesn't specify a way to notify apps of new spaces. If your app creates a space, you can track it via Sap.TrackSpace. However, if a new space was created in another app or the user was added to a space by another space host, there isn't a prescribed way to discover it. We kicked off a discussion on the Community Discourse around discovery of permissioned spaces. In this side-page, we outline the problem and a possible solution further.

On a possible lexicon to maintain discoverability with permissioned spaces.

Problem statement

Permissioned spaces poses a new problem for apps trying to be interoperable: discovery of new spaces apps may want to show users. With public data this is not an issue because all apps can see, and hence discover, all data. The following scenario outlines the problem we are discussing:

Bob & Alice are two friends with ATmosphere accounts (PDS'es) and they each use their own group chat app. Alice uses ChatMe and Bob uses LetsChat. ChatMe and LetsChat coordinate to use the same lexicons for interoperability, and use example.groupChat as the space type for a private, access-gated, group chat. Bob & Alice are not yet chatting with each other.

Current state of Alice & Bob:

At some point, Bob decides to start a group chat and invite Alice to it. Bob's chat app, LetsChat, adds Alice to the example.groupChat space. Bob now sees Alice as part of the group in his app.

But how does Alice's app know to show Alice the new group chat she just got added to?

This seems like it would be a common pattern for apps wanting to interoperate with each other, for e.g. Roomy & Colibri. The problem falls out of the fact that it is easy for Alice's PDS to know what spaces she has written to, via apps authenticated with her PDS, but it is difficult for Alice's PDS to know what spaces she has been added to, or has read access to.

This problem has come up in the Fediverse / other federated and open networks, so hoping to do additional research on prior art here and what good patterns were. The high-level observation is that Alice needs some sort of 'inbox', or way to receive notifications. There's been a couple of good threads about this (does the end-game of ActivityPub and atproto look the same?) that I can't find now that I need it.

Lexicons for space discovery?

Alice's app ChatMe needs to discover group chats that Alice is being added to initiated by other applications. Only ChatMe or other apps Alice has given access to should be able to see who is adding Alice to new spaces, but (possibly) arbitrary apps that Alice's friends use should be able notify Alice/ChatMe about new spaces that may have chats she wants to see.

One way this could be implemented is with a dedicated space, say at://<user>/space/example.space.discovery/self that is hosted by the user's PDS and is write-open (anyone can write) and read-closed (only Alice can read), just like an email inbox is. As @dholms.at noted in the Discourse post, this space would likely become a spam vector, which is one of the major downsides to introducing any inbox-shaped concept to the PDS.

For the purpose of Habitat, organizations (which are space hosts) can coordinate independently to connect with each other and share new spaces. For example, the admin of org #1 could send a request to the admin of org #2 which on acceptance, makes both space hosts aware of each other, and they can cooperate to notify each other of new spaces that involve the other. We haven't fleshed this out fully, but we will definitely need some sort of protocol around this. In the organizational data server we are building, orgs will need to interoperate with each other and add each other to spaces when they don't necessarily use the same apps to interact with. We want to make it possible to add "eng team from other org to this chat channel" or "brand team from other company to this doc" in an ~ATmospheric way~.

However, this feels like a real need that not just we'll have but also app developers in the broader ecosystem, in order to keep the ethos of "permissionless interoperability" going. We would love to collaborate with folks to see if there can be standards defined around spaces discoverability so if folks have other ideas around this, please reach out! Otherwise, more on this coming 🔜!

Sap has a lot of moving parts like auth sessions, crawl state, notification registration, resync loops, etc. Hence, Sap needs good observability for when something does goes awry. It currently emits Opentelemetry metrics and traces, but we're also planning on adding an API to introspect Sap's internal state and maybe also a UI to visualize it.
Sap should theoretically be compatible with other permissioned spaces implementations like Nick Gerakines' and HappyView except we're using network.habitat.* namespace. We'll experiment with these implementations and try to make Sap work with them as well.
Proper versioning for future sap releases.

We are testing sap with teams building applications on top of Habitat. If you decide to use sap, please join us on our Discord or email us at hello@habitat.network to give us feedback and help us improve it! Godocs are here: https://pkg.go.dev/github.com/habitat-network/habitat/pkg/sap

What's next:

- We have some very exciting releases coming the next two weeks, including finer-grained permissioning on top of spaces, and management of spaces across apps. Follow us @habitat.network to keep up!

Until next week!
- Habitat team 🌱

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

building [at] habitat @habitat.network