Tag

Databases

Articles and publications tagged Databases across the Atmosphere.

16articles6publications

Articles

Publications

Redowan's ReflectionsApr 11, 2026

Dynamo: Amazon's highly available key-value store

Key takeaways from Amazon's 2007 Dynamo paper.

Distributed SystemsDatabases

Isaac's RamblingsFeb 6, 2026

On database migrations and workflow friction

Database development workflows are slow, fragile, and frustrating despite clear data models.

databasesentity-framework

MacKuba blogOct 15, 2025

How I ran one Ruby app on three SQL databases for six months

Since June 2023, I’ve been running a service written in Ruby (Sinatra) that provides several Bluesky custom feeds (initially built with a feed for the iOS/Mac developers community in mind, later expanded to many other feeds). If you don’t know much about Bluesky feeds, you make them by basically running a server which somehow collects and picks existing posts from Bluesky using some kind of algorithm (chronological or by popularity, based on keyword matching, personal likes, whatever you want), and then exposes a specific API endpoint. The Bluesky AppView (API server) then calls your service passing some request parameters, and your service responds with a list of URIs of posts (which the API server then turns into full post JSON and returns to the client app). This lets you share such feed with anyone on the platform, so they can add it to their app and use it like any built-in feed. (If you’re interested, check out my example feed service project.) In order to provide such service, in practice you need to connect to the Bluesky “firehose” streaming API which sends you all posts made by anyone on the network, and then save either those which are needed for your algorithm, or save all of them and filter later. I chose the latter, since that lets me retry the matching at any time after I modify the keyword lists and see what would be added after that change (and also some of the feeds I now run require having all posts). I also use the same database/service to generate e.g. the total daily/weekly stats here. All posts made on Bluesky is much less than all posts on Twitter, of course, but it’s still a lot of posts. At the moment (October 2025), there are around 3.5M posts made on average every day; at the last “all time high” in November 2024, it was around 7.5M per day. A post is up to 300 characters of (Unicode) text, but since I also store the other metadata that’s in the record JSON, like timestamp, reply/quote references, embeds like images and link cards, language tags etc., it adds up to a bit less than 1 KB of storage per post on average. In addition to that, the firehose stream (if you use the original CBOR stream from a relay, not Jetstream, which is a JSON-serving proxy) includes a lot of overhead data that you don’t need in a service like that, plus all the other types of events like handle changes, likes, follows, blocks, reposts, and so on. The total input traffic is around 15 Mbit/s average right now in October 2025 (or around 5 TB per month), and it used to be around twice that for a moment last year. (Jetstream sends around an order of magnitude less, especially if you ask it to send filtered data, e.g. only the posts.) On disk, the millions of posts per day add up to a few gigabytes per day. Since I was running this on a VPS with a 256 GB disk (Netcup, RS 1000 – reflink), I have a cron job set up to regularly prune all older posts and keep only e.g. last 40 days worth of them (since I don’t really need to keep the older posts …

RubyDatabases

Devtools FMDec 2, 2024

Anselm Eickhoff - Jazz Tools

Anselm Eickhoff presents Jazz Tools, reimagining client-server boundaries for local-first apps with collaborative data structures.

typescriptfrontend

Build.msOct 23, 2024

Monarch: Minimal, Manageable, Migrations 🦋

Introducing Monarch, a simple yet flexible migrations library for Swift that helps manage user data changes across app updates with minimal hassle.

DatabasesEngineering

Devtools FMSep 8, 2024

Predrag Gruevski - Trustfall, Cargo Semver Checks, and the Future of Query-Based Tools

Predrag Gruevski explores Trustfall, a revolutionary query tool that turns anything into queryable data, and cargo-semver-checks for Rust semantic versioning.

rustrustlang

Tedium: The Dull Side of the InternetSep 3, 2024

The Worst Form On The Internet?

Oracle’s form to access free cloud server space seems designed to discourage you from taking advantage of the offering. It’ll leave you frustrated.

oracleoracle cloud

Devtools FMJul 7, 2024

James Arthur - ElectricSQL

James Arthur from ElectricSQL explains how their local-first sync engine transforms Postgres into a distributed database that runs locally on every device.

technologyreact

Tedium: The Dull Side of the InternetApr 27, 2024

Your Phone’s Other Number

Where did the IMEI number come from, why is it so freaking long, and why does your phone need it? If you’ve ever wondered, we’ve got you covered.

imeiserial codes

Devtools FMFeb 19, 2024

Glauber Costa - Forking SQLite and Building a Distributed Database with Turso

Glauber Costa explains how Turso forks SQLite into libSQL to create a distributed database, drawing from his Linux kernel development experience.

techsqlite

Tedium: The Dull Side of the InternetOct 27, 2023

Spooky Database Relations

A discussion of the least-on-brand thing to ever happen on Halloween: The Halloween problem, a database querying issue with a long legacy.

databasessql

Build.msJun 22, 2022

Model View Controller Store: Reinventing MVC for SwiftUI with Boutique

Introducing Boutique, a new architecture that reimagines MVC for SwiftUI by adding a Store layer to handle state management and data flow in a familiar, approachable way.

DatabasesEngineering

Tedium: The Dull Side of the InternetJun 17, 2022

We Value Your Feedback

How the suggestion box, once a simple tool for giving feedback, played a role in the weirder and darker data-hungry present for many companies.

suggestionssuggestion box

Tedium: The Dull Side of the InternetSep 20, 2016

The Big Data Jukebox

When AllMusic launched 25 years ago, it wasn't an obvious big data play. But it became one. Hidden in its millions of entries is music's collective history.

pop culturemusic industry

MacKuba blogOct 10, 2010

Notes from the MongoBerlin conference

This week I went to Berlin for 2 days to attend the MongoBerlin conference organized by 10gen, the creators of MongoDB. I took a lot of notes from the presentations, and I figured this may be useful for someone if they missed a presentation (or just couldn’t go to the conference). Feel free to correct me if I got something wrong :) “BRAINREPUBLIC: A high-scale web-application using MongoDB and RabbitMQ” MongoDB vs CouchDB: CouchDB - slower (because it uses a HTTP API), easier replication MongoDB - faster they needed high performance, so they chose MongoDB Mongo has a rich query API that lets you do more than Couch’s map/reduce they use RabbitMQ, Solr, Varnish proxies and Heartbeat MongoDB is a swiss knife of NoSQL, very fast, reliable and very easy to use NoSQL doesn’t mean you can be lazy about proper application and database design current problems with MongoDB: quite slow replication speed (someone from the audience said that replica sets are fast) indexes can be very slow if they don’t fit into memory simple authentication - sometimes too simple map/reduce is single-threaded (locks the machine) “MongoDB Internals: How It Works” what does an insert do: constructs Object ID (includes timestamp, machine/process ID, counter); two clients at the same time can’t generate the same ID, and IDs always increase with time, so order by id = order by time converts data to BSON sends a TCP message (using MongoDB Wire Protocol) query: returned as a cursor: first n results + cursor id, and then you can get next results using the cursor (n depends on the size of documents) contrary to relational databases, cursors are not associated with sockets, and can be shared between connections; server cleans up cursors periodically if you don’t use them (this gives more flexibility in connection management on the client side) authentication - should rarely be required, it’s recommended to protect the database at the network level (using firewalls) query optimizer: it’s empirical, i.e. at first it tries all possible ways to get the results, and then remembers which one works best (it runs all algorithms in parallel and finishes as soon as one of them finishes), then reuses that knowledge in future requests if the selected algorithm becomes very slow, it tries all possible ways again so first time a query is called, it might be quite slow on the other hand, if something changes later, e.g. an index becomes slow, Mongo will work around that commands: return short values as results, not actual data (there’s a limit on how much data you can return from a command) internally, they’re performed as finds on $cmd collection all commands can be invoked via runCommand with the name as one of parameters safe mode: normally, commands return as soon as the request is written to the socket if you want to wait until it arrives on the other side and is executed, you use safe mode it uses getlasterror command, which waits until the operation is executed, and returns an…

Databases

MacKuba blogSep 2, 2009

Installing Sphinx on MacOSX from ports

A few days ago I spent a couple of hours trying to install the full text search engine Sphinx on my work computer in order to use it in my current project. I’m posting the details here, maybe this will save someone some time… Firstly, if you’re like me and you look for some kind of binary package (like .dmg) first – sorry, there isn’t one. This leaves us two options: compiling Sphinx manually from the source, or using a package manager like MacPorts. I always avoid the first method if I have any other option, so I tried the ports. Here’s the first problem: Sphinx requires MySQL, and while I have it on the disk of course, I’ve installed it from a dmg package, not from ports – which means it’s in a different path than ports expect and they have no idea where that is. I got scary errors like: $ sudo port install sphinx ---> Building sphinx with target all Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_textproc_sphinx/work/sphinx-0.9.7" && make all " returned error 2 Command output: sphinx.cpp: In member function 'virtual const char* CSphSource_MySQL::SqlError()': sphinx.cpp:9397: error: 'm_tMysqlDriver' was not declared in this scope sphinx.cpp:9397: error: 'mysql_error' was not declared in this scope sphinx.cpp: In member function 'virtual bool CSphSource_MySQL::SqlConnect()': ... So I had to override the options that ports pass to ./configure; the ports' man page wasn’t very helpful about how to do it, but luckily Google was. To do that, you need to pass the options as configure.args option to the port command. Here’s what I did (I had to clean the port first, otherwise it wouldn’t repeat the configure phase): $ sudo port clean sphinx ---> Cleaning sphinx $ sudo port install sphinx configure.args="--with-mysql-includes=/usr/local/mysql/include/ --with-mysql-libs=/usr/local/mysql/lib/" ---> Fetching sphinx ---> Verifying checksum(s) for sphinx ---> Extracting sphinx ---> Configuring sphinx ---> Building sphinx with target all ---> Staging sphinx into destroot ---> Installing sphinx 0.9.7_0 ---> Activating sphinx 0.9.7_0 ---> Cleaning sphinx Voila. Let’s start ThinkingSphinx: $ rake thinking_sphinx:start Sphinx 0.9.7 Copyright (c) 2001-2007, Andrew Aksyonoff ERROR: malformed or unknown option near '--pidfile'. usage: searchd [--config file.conf] [--console] rake aborted! WTF? Again, Google to the rescue. It seems that the --pidfile option which is used by ThinkingSphinx was added only in version 0.9.8 of Sphinx… but MacPorts doesn’t have that version. Or does it? The MacPorts project page for Sphinx shows that it should be at 0.9.8. So why am I getting 0.9.7? After a next session of googling I figured out that I may need to upgrade the ports system itself – apparently the newest version of the Sphinx port was somehow incompatible with the older version of ports. MacPorts can be updated with the command port selfupdate: $ s…

MacRuby/Rails

You've reached the end.