Topics on Bluesky

Towards the end of 2024, I was sitting at the office working on something boring when Paul (my manager at the time) said to me "you can build trending topics, right?" I looked back at him and took a moment to think. The first answer that came to mind was "no", because I mean...what? I didn't really have much idea what on earth could go into building a trending topics system. However, I of course did not say no and instead said "yeah sure, why not?"

We had some initial discussions on what we even wanted out of the system. We were okay up front with the system being very rudimentary. Some initial ideas revolved entirely around the use of hashtags. I felt that was extremely lackluster and even though I didn't walk away saying "no" to that idea, I knew in the back of my mind that I wasn't going to let that be the way I implemented this. There must be some better way to make this work, right? Hashtags are cringe, they would never work for this sort of system yet alone would anyone use them. On top of that, not to mention they'd just rapidly get abused.

And so, I set out on a little adventure of figuring out how to build this - in some two-week timeline that I was given, if I remember correctly (maybe it was three) - and put together a few different pieces of a system that on their own were pretty useless, but together actually work quite well.

So if not hashtags...

The first thing that I knew we needed to figure out was what we could use to actually find trends that were happening on the network. We knew that hashtags were one mechanism that we could use, but those on their own wouldn't work too well. We also couldn't - especially at the time - just feed all the posts into some LLM and have that decide what "groups" a post would go under. Not only was I afraid that would struggle pretty significantly to do proper grouping, but it was super cost prohibitive. There are also a lot of finetunes of BERT for example out there which are useful at clustering based on broad topics, but nowhere near as specific as we would like them to be.

Something else that came to mind was to rely on embeddings of posts. If we started embedding all of the posts and finding sudden growth in similarity in some little region of the network, that might help us pull out a trend. The problem here is that it's pretty difficult to take just the raw text of a post, group them nicely together, then figure out what posts are all about. Posts that are all about the same topic might have significant differences in how the rest of the post is formulated, expanding the distance between the posts that are actually talking about the same thing. Likewise, posts that are totally unrelated can actually be semantically similar, and as a result you end up with groups that really might have no meaning at all.

After some research, I came across the idea of Named Entity Recognition. NER is a natural language processing (NLP) technique that identifies and classifies named entities in a given piece of text into a set of predefined categories. For example, let's look at the sentence "Yesterday, Daniel Holmgren announced Bluesky's new tool called Tap, which lets you easily backfill the ATProtocol network." The following entities may be extracted from that text:

- Yesterday -> DATE
- Daniel Holmgren -> PERSON
- Bluesky -> ORG
- Tap -> PRODUCT
- ATProtocol -> PRODUCT

Immediately, we have reduced the "surface area" that we're working with significantly, which in turn makes it significantly easier to decide what a given thing is about. No longer do we have some random sentence without meaning (or hashtags...), but a sentence that we can actually determine the topic of.

spaCy

Now that we had a sense of direction, it was time to figure out how to do it. One of the common things you'll find looking online for an option for NER is a collection of NLP finetunes, like dslim/bert-base-NER. I started playing with this and other models (including the large finetune of dslim/bert-NER) and was left not feeling too great. For example, let's take the example sentence above and see what both the base and large versions of that BERT finetune give us.

With the base model:

- Daniel Holmgren -> PERSON
- Bluesky -> PERSON
- Ta -> MISC
- ATProtoc -> ORG

And with the large model:

- Daniel Holmgren -> PERSON
- Bluesky -> ORG
- Tap -> MISC
- ATP -> ORG

In the first example - the base version - we see that the model was successfully able to find the items of interest in the input text, however it has numerous problems. First of all, Bluesky here was determined to be a person (for what it's worth, for our use case this wouldn't have been the worst thing ever). The more obvious problems are the model cutting Ta out of Tap for the product, and ATProtoc for the ATProtocol. This obviously wasn't going to work.

The large model comes with much better results. It correctly identifies Bluesky as an organization now, and the full name Tap for the product. It still struggled to correctly extract ATProtocol though.

But it gets worse, depending on the finetune you're working with. Some finetunes use bert-cased and others use bert-uncased. These examples were using bert-cased, and as a result have very poor results with uncased sentences, something that's obviously common on Bluesky. See this result from the large model when removing casing from the sentence.

holm -> PERSON

We could switch over to an uncased model and get better results again, but still with errors like here in atprotocol.

- daniel holmgren -> PERSON
- bluesky -> PERSON
- tap -> ORG
- atp -> MISC
- rotoco -> ORG

I was feeling a bit let down by these options, and there were already some other limitations that I was anticipating coming when I stumbled upon spaCy. spaCy is a NLP library that additionally comes with a set of pretrained models for doing not just NER but a range of other things like part-of-speech tagging (nouns, verbs, adjectives, etc.), dependency parsing, and lemmatization. I did some testing, and I quickly found that the entity results that spaCy was giving me were far better than those from the BERT-based models I had been trying previously.

import spacy


SENTENCE = """
Yesterday, Daniel Holmgren announced Bluesky's new tool called Tap, which lets you easily backfill the ATProtocol network.
"""


def main():
    nlp = spacy.load("en_core_web_trf")
    doc = nlp(SENTENCE.strip())

    print("Results with casing:")

    for ent in doc.ents:
        print(f"- {ent.text} -> {ent.label_.upper()}")

    doc = nlp(SENTENCE.strip().lower())

    print()
    print("*" * 60)
    print()


    print("Results without casing:\n")

    for ent in doc.ents:
        print(f"- {ent.text} -> {ent.label_.upper()}")

if __name__ == "__main__":
    main()

🎗️ ❯ uv run main.py
Results with casing:
- Yesterday -> DATE
- Daniel Holmgren -> PERSON
- Bluesky -> ORG
- Tap -> PRODUCT
- ATProtocol -> PRODUCT

************************************************************

Result without casing:
- yesterday -> DATE
- daniel holmgren -> PERSON
- bluesky -> ORG
- tap -> PRODUCT

As you can see, we're immediately starting to get better results. When using a sentence with proper casing, the spaCy model accurately extracts all of the entities that we'd expect. Without casing, we still get very close - and close enough to do what we want. If we were trying to track a trend with this particular sentence, the two things that are most interesting - Daniel Holmgren and Tap were both accurately found, and both given the correct tags of PERSON and PRODUCT. With reliable entity extraction in hand, we could move on to the next challenge.

Starting to aggregate

Now that we had one of the two most difficult parts of the problem out of the way, we could move on to start aggregating posts into "topic" groups. Each time a top-level post would come through the firehose, we'd take the post's text (and any post that it may have been quoting) and use spaCy to extract the entities from the text. This is extremely performant when running the en_core_web_trf model on GPU and was easily able to handle the throughput of the firehose.

After gathering our list of entities for the given post, there were a few other things we wanted to take care of. The first thing I noticed was that there was a lot of worthless information being given back to me. Notice in the above example that spaCy extracted yesterday as an entity. While correct, we generally do not care much about dates on their own. There are also a lot of other types of entities that we do not care about which we want to ignore.

PERSON:      People, including fictional.
NORP:        Nationalities or religious or political groups.
FAC:         Buildings, airports, highways, bridges, etc.
ORG:         Companies, agencies, institutions, etc.
GPE:         Countries, cities, states.
LOC:         Non-GPE locations, mountain ranges, bodies of water.
PRODUCT:     Objects, vehicles, foods, etc. (Not services.)
EVENT:       Named hurricanes, battles, wars, sports events, etc.
WORK_OF_ART: Titles of books, songs, etc.
LAW:         Named documents made into laws.
LANGUAGE:    Any named language.
DATE:        Absolute or relative dates or periods.
TIME:        Times smaller than a day.
PERCENT:     Percentage, including ”%“.
MONEY:       Monetary values, including unit.
QUANTITY:    Measurements, as of weight or distance.
ORDINAL:     “first”, “second”, etc.
CARDINAL:    Numerals that do not fall under another type.

For example, QUANTITY, ORDINAL, and CARDINAL are all things that we do not need to worry about, as are DATE, TIME, and MONEY. By ignoring these, we again significantly reduce the "surface area" of what we have to track and worry about.

We also naturally came across a wide range of topics that we simply did not want to surface. This ranged from things like harmful content (i.e. slurs) to non-age-appropriate topics (i.e. NSFW content). Because Bluesky does have a large amount of the latter, we were quickly able to find the common terms that would come up (you don't want to know) and added them to a handwritten exclusion list. This wasn't going to be enough for broader safety concerns, but it was enough for now to get us closer to the mark on what we wanted to track.

Filtering complete, we would finally:

• Create a new topic in the Postgres database or increment the posts count for each of the named entities that we got from the text
• Store the post itself by its AT-URI in Postgres
• Update the join table between that post and its assigned topics

With all of this out of the way, the only thing left for us to do for tracking purposes was to continue updating the Postgres database with interactions that appeared on the firehose. Each post row in the table included columns for interaction counts (i.e., likes, reposts, replies) that would get incremented at a set interval based on the interactions those posts received. We were finally ready to start modeling how to find new trends on the network...

Finding the trends

Once we began to gather the aggregate data per entity in a post, the next thing we wanted to do was make sense of that data. An initial problem to consider is that seeing a bunch of posts about, say, "New York" doesn't really do much to help you. There are a bunch of different things that could result in that particular entity having chatter. Maybe it's the New York Knicks, maybe the Jets, or maybe it's the New York mayoral election...

To resolve this, the first thing we wanted to do was establish a threshold for when we considered a particular topic to actually become trending. General chatter about an entity is always happening, especially for broad things like locales or people. Playing with data over the course of a few days was enough to determine a "baseline" for the amount of chatter required for a given entity to potentially be considered as a new discussion point.

Second, we needed to find out if the chatter is actually about a particular event that was going on. For this, taking the top N posts for a given entity and having an LLM decide if those posts were closely related was a great tool. If the "New York" entity had 20 posts that were all about different things happening in New York, this was not considered to be a trend. However, if the bulk of those posts were about, say, the election, we could then confidently make the determination that this was a real trend.

Using an LLM for this step also gave us an additional advantage. "New York" is too broad of a label for a "trend". What we really wanted was "New York Election". Unfortunately, pulling this context out of the data from just NER alone wasn't enough, so we'd instead ask the LLM to return not only a determination on if a group of posts constituted a trend, but also what the trend was about. We were additionally able to request that a "broad" topic be assigned to the result, such as politics, sports, or gaming.

This came with caveats, especially in 2024 when we originally launched trends. Letting the model have freer rein on labeling a topic - say, by asking for a 1-2 sentence explanation - led to some, frankly, horrific results. Quite frequently, the LLM would return some entirely incorrect description, including claims that "person N has passed away". Some early evaluations immediately looked too concerning to use for user-facing results, so we ruled out offering longer explanations. Even today, it is still common to see other websites such as news organizations or other social media apps include descriptions of a topic that are outlandishly irrelevant or simply incorrect - often with hilarious or even grim effects.

Instead, we asked the LLM to return a 2-3 word title for the topic, restricted to some specific number of characters. Simply "asking" the LLM to do this didn't give us great guardrails against any violative responses of that rule, but it did keep the LLM from giving us the worst results. Additional code-based validation also prevented anything from accidentally making its way into the user-facing results that we didn't want.

By this point, having this pipeline run at a consistent and frequent enough interval to continually surface relevant trends was working pretty well, and we were getting quite close to the results we wanted. There were a few additional considerations though that became relevant over time...

More Guardrails and Curation

I was originally letting this system throw me results into a private Slack channel that I could review every so often. Things slowly started looking more and more promising, and we were getting close to something that we could productionize and make available for public viewing. There were a couple of concerning things though.

For one, we needed to be very careful about both violative content (spam abuse, illegal topics, or content that was not suitable for all audiences) making its way into the public results. Our initial prompt that gave us an evaluation on whether a topic was truly a trend or not requested the LLM to also consider whether the posts given to it violated any of the rules we had set in place. Unfortunately, this alone wasn't quite enough. While the LLM was successful at preventing outright illegal content from surfacing (be it from our prompting or from the LLM's own guardrails preventing the processing of that data), it didn't do a great job at preventing some content that was obviously not suitable for all audiences from making its way in - especially whenever that content was centered around particularly niche communities or vague descriptions.

To solve for this problem, we added a few additional passes to our pipeline. Once a topic successfully was found and returned from our initial LLM prompt, we would next do two things. First, we'd once again ask the LLM to do a sanity check on its own output, which consisted of the generated topic name and a few-sentence long summary of the posts. Our prompt here was a good bit more descriptive in what we were looking for, and there was a significant reduction in topics that violated our initial set of rules.

Second, we would once again take our top posts for the given topic and use a moderation-specific API that we were already using elsewhere in our stack to determine if anything inside of each post was violative of any of our criteria. This became the missing piece that finally yielded clean results, and was definitely the most important part of the pipeline that made the results truly good for using in public-facing results.

On the curation note, the other problem we were seeing was topics being created for things that had already become topics. Again using New York as an example, there would often be situations where "Jets" would become a topic alongside "New York". Both would be referencing the Jets, but because these were two distinct entities that spaCy was giving us, they were clustered separately. We needed a way of "merging" these topics into a single topic.

Once again, we added two additional pieces to get the results we wanted. First, we would use an LLM with a prompt including the currently trending topics alongside the posts, summary, and topic name the LLM had assigned in a previous step. We asked the LLM to decide if this topic was similar to another existing topic, and if so, we'd strongly consider the two topics a candidate for a merge.

Additionally, we would take embeddings of the content of the top posts in the new trend, and compare them to the embeddings of top posts in the existing topics. While this had variable results, it did give us additional good signal if the LLM wasn't quite certain what to do in the first step.

At last, with merges out of the way, we finally had a set of trends that we could surface in the app.

A Saturday Hack

At this point, we jump to the actual launch of trending topics. When Bluesky initially launched topics, we decided to do what other apps at the time (still today?) were doing for topics - just redirect the user to the search page with the term set to whatever the topic was. Seems simple and easy right? Well...

Obviously, on the surface this worked okay for directing the user to the relevant topic, but what it didn't reliably do was surface the actual posts that had even warranted the trend in the first place! Call this a failure of Bluesky's search function if you will (that's certainly an argument), but unfortunately in many cases getting redirect to the topic about New York would result in either the "Top" results which often were posts from hours or even days ago or the "Latest" results which - while more relevant - were very often just...slop.

Naturally, I got nerd sniped on a Saturday morning while looking at the results for a topic that by name had captured my interest. I started scrolling through though and was quickly left sad by the results. The top search results were simply not giving me any context about what was actually happening. Extremely irrelevant...

Now, when we initially designed the API for fetching and displaying topics within the application, we had the idea of possibly having feeds, starter packs, or even a specific user be displayed in the results. As such, we also wanted to be able to redirect users to those feeds, profiles, or otherwise. So...what if we just started directing users to a feed for a topic rather than search? We already had an index of the posts for a topic (or the merged topics), the relevant counts for those posts that we could use to drive decent enough ordering for the items in a feed, and moderation handled by the Bluesky labeler itself...so what if we just made feeds from that data instead of relying on search?

And thus, my Saturday morning was spent adding a feeds API to the topics service. Each time that a new topic became a trend, we would create a new feed on a Bluesky-operated account. Once created, we would use the URL of that feed as the path given in the API responses for users to get directed to and populate the feed with the posts we were placing into the topic. Immediately, the quality of the trending experience improved dramatically. In fact, the early results that we were seeing were so good that I decided to simply switch out search with these new feeds that same day!

And that's more or less where we are today. What started as a "sure, why not?" turned into one of the more interesting things I got to work on. Not necessarily because of complexity, but rather how putting together a number of different pieces led to a fairly well functioning experience. NER to reduce surface area, LLMs for trend detection and labeling, embeddings for merge candidates, and dynamically generated feeds to surface the content people care about and is actually interesting.

There's still plenty of room for improvement. Non-English support doesn't yet exist, but is feasible as spaCy has models for each of the languages that is popular on Bluesky. The merge logic can occasionally be hit-or-miss - sometimes too aggressive, though more frequently not aggressive enough (especially around sports topics). And there are edge cases where the system surfaces something that... probably shouldn't have made it through. But for something built in a two-ish week sprint and rarely iterated on since, I'm pretty happy with where it all ended up landing.

And as for cost...well, the entire pipeline can conceivably cost under $100 per year, depending on the infrastructure you use and how you break it down. For example, if you wanted to use OpenAI APIs for the LLM and moderation API then run spaCy on your own, you could easily land in the under $100 a year range, assuming you already have at your disposal a consumer-grade GPU (like a 4090) to run the spaCy model on (yes, you can handle the full throughput of the Bluesky network in real time with a single instance of spaCy running...)

With that in mind, there's a lot of potential here for creating even more narrowly defined feeds using NER alongside the public Wikidata database. Indeed, there's already ways of turning the Wikidata info into a database that spaCy can use alongside its NER functionality, which deserves a post on its own...Combine that with something like a locale-based feed, a sports team fan feed, or a finance feed and there are some really interesting things you could do! Maybe one day I'll find the time to give it a real shot...