Real stories

Every page on this site explains a mechanism. This page shows what happens when the mechanism meets real traffic. Sixteen stories, all from companies that wrote up what broke. Each one has the numbers, the fix, and the lesson I took from it. The source is linked at the end of each story.

None of these are my incidents. I read them the way I read a postmortem: what broke first, what the numbers were, and what decision would have prevented it.

PostgreSQL · ten stories

PostgreSQL: the layout is the cost

Most of these are the same failure wearing different clothes: a physical layout decision, made early, priced later at scale. Adyen and Figma show it at the far end, where the partition and the shard key decide everything before the first query runs. One is a planner that had the right index and refused to use it. GitLab and Instacart are the exceptions: a backup nobody tested, and a split by domain instead of by shard key.

PostgreSQL

Uber: one update became four writes, and the WAL crossed an ocean

Uber started on Postgres. One monolithic backend, one database holding the trip data. As the company grew, the database became the constraint. The post names five limitations: inefficient writes, inefficient replication, table corruption, poor replica MVCC, and hard upgrades. The first two are the expensive ones, and they share one root cause.

One logical update, four physical writes

1 logical UPDATEone row changes
→
new row versionin the heap
index 1rewritten
index 2rewritten
index 12rewritten
primarywrites the WAL
→
physical WAL streamevery extra byte crosses the network
→
replica, other data centerreplays it all

A table with a dozen indexes rewrites all twelve on every update. The cost is physical, not logical.

They were running Postgres 9.2. Every update wrote a brand-new row version, and every secondary index pointed at the row's physical spot on disk. One logical update became 4 or more physical writes. A table with a dozen indexes rewrote all twelve on every update. The WAL stream was physical too, so every extra byte crossed the network to replicas in the other data center. A bug in 9.2 returned duplicate rows on replicas after a timeline switch, and the only fix was a full resync. Long queries on replicas got killed by cleanup. Upgrades needed full downtime, so old instances stayed on 9.2 while 9.5 was current.

The fix was Schemaless, built on MySQL and InnoDB. In InnoDB, a secondary index points at the primary key, not at a disk address. An update touches the row in place and leaves the other indexes alone. Replication is logical, through the binlog, so the stream across data centers is far smaller.

Write amplification went away. Replication got cheaper. Upgrades stopped needing full downtime. The lesson I take is not that Postgres is slow. The lesson is that index layout and replication format are the whole cost at write-heavy scale. One more thing: the post is from July 2016, about Postgres 9.2. Later versions fixed some of the named bugs. The layout difference is still real.

PostgreSQL

Notion: VACUUM stalled, and the database nearly stopped writing

Notion ran its product on one Postgres monolith. Every paragraph, heading, image, and to-do is a block, and every block is a row in one giant table. By mid-2020, usage had outgrown the single database.

Before and after: one monolith, 480 shards

Before

One Postgres monolith. Every block is a row in one giant table. VACUUM stalls, dead tuples pile up, and transaction ID wraparound threatens to stop all writes.

After

480 logical shards, 32 physical databases. Routing by workspace ID, so one workspace's data stays together.

32 physical databases, 15 logical shards each. The shard key is the whole project.

500 GBCap per table, set by the team before the migration.
10 TBCap per physical database.
60K IOPSCap per physical database.
5 minCutover downtime, after a 3-day backfill on a 96-CPU machine.

The trigger was specific. VACUUM started stalling, so dead tuples stopped being reclaimed and disk space stopped coming back. Behind that sat transaction ID wraparound. Postgres stops all writes rather than overwrite old data. That is not a slowdown. That is the product stopping.

The design was application-level sharding, routing by workspace ID so one workspace's data stays together. They built 480 logical shards spread across 32 physical databases. They set their own caps: 500 GB per table, 10 TB per physical database, 60K IOPS. Logical replication could not keep up with the block table's write volume during the first snapshot, so they moved to an audit-log double-write. The backfill ran 3 days on an m5.24xlarge with 96 CPUs. They used dark reads to verify, and the verification code was written by different people than the migration code. The cutover took 5 minutes of scheduled downtime.

Their own lessons: shard earlier, aim for zero downtime, and use a combined primary key instead of a separate partition key. Mine: the shard key decision is the whole project. Workspace ID kept queries local. Pick the key from the query pattern, and pick it before you are forced to.

Two details from the same post. They prepared and tested a reverse audit log, so they could replay sharded writes back to the monolith if they had to roll back. They never needed it. They picked 480 shards because the number has many factors, so they could grow from 32 to 40 to 48 hosts and keep shards even; with 512 they would have had to double from 32 to 64. They used separate tables per schema, schema001.block and so on, instead of one natively partitioned table per database, so all routing stays in the application.

PostgreSQL

Notion: the re-shard, 32 databases to 96, with zero downtime

Part 2 of the Notion sharding story above. The 5 minute cutover and the 3 day backfill belong to the first sharding, not this one.

32 to 96Databases. Same 480 logical shards, 15 schemas each down to 5.
3 days to 12 hoursSync time, after skipping indexes during the copy and rebuilding them after.
~20%CPU and IOPS at peak after the move, down from over 90 percent on hot shards.

The problem

32 databases running hot. Some shards passed 90 percent CPU at peak. Many near full on provisioned disk bandwidth. PgBouncer hitting connection limits. A new year spike was coming. Out of headroom. The database was not broken. It was full.

32 to 96, and the pool split first

Before: 32 databases

15 schemas each. Hot shards over 90 percent CPU at peak. PgBouncer hitting connection limits.

After: 96 databases

5 schemas each. Same 480 logical shards. CPU and IOPS around 20 percent at peak.
PgBouncer group 124 databases
group 224 databases
group 324 databases
group 424 databases
CPU on hot shards, before
over 90%
CPU at peak, after
about 20%

When you reshard, reshard the pool too. The connection math is what bites you.

The solution

No bigger machines. Tripled the fleet, 32 to 96, same 480 logical shards, 15 schemas per database down to 5. Smaller instances and disks to hold cost down. Disk was not the bottleneck. Copy with Postgres logical replication: 3 publications per old database, 5 schemas per publication. Indexes skipped during the copy, rebuilt after. Sync time fell from 3 days to 12 hours. The pool nearly broke the plan. About 100 PgBouncer instances, up to 6 connections per shard, 600 per shard. A naive cutover would push that to 18 per old shard. Cutting connections made queries back up in testing. So PgBouncer was sharded first, into 4 groups with 24 databases each. That also limited a PgBouncer failure to 25 percent of the fleet. Failover: pause traffic in PgBouncer, let server connections finish, check replication caught up, flip the mapping, reload PgBouncer, revoke the old login, reverse replication for rollback. Checkpoints and rollback at every step. Full process in staging first, tested multiple times. Dark reads checked the copy first: parallel reads to the follower, queries returning up to 5 rows, small sample, 1 second pause for replication to catch up. Near 100 percent equivalence. Outcome: no observed user downtime. Worst case about 1 second of saving spinner. After the move, CPU and IOPS sat around 20 percent at peak.

Let's talk about the design

Notion picked tripling over bigger machines because bigger machines would hit the same wall later, and one big box cannot be tuned per shard. PgBouncer got sharded first because the connection math was the real limit.

Cost: more machines to run, on smaller instances.
Cost: hot workspaces stand out more when the load spreads. A few spaces drive outsized CPU.
Cost: the rollback plan cost extra work, reverse replication at every step.

The part I would copy is sharding the pool first. The connection math is what bites you.

Read this when your fleet is hot and out of headroom.

PostgreSQL

Notion again: the index existed, and the planner ignored it

Same company, a different problem. Notion ran one large Postgres database on RDS from 2015 to 2020. A new feature was already 20 percent of the application's query load before it shipped to everyone. One of its queries filtered JSON on space_id and permission.

Same query, two index shapes

GIN index existedon space_id and permission
→
planner skipped itpoor cardinality, stale statistics
→
jsonb_path_ops GINfits containment queries on JSON
Before, default GIN shape
about 5000 ms
After, jsonb_path_ops GIN
about 600 ms

More than 8 times faster. The index existed. The planner ignored it. The fix was the index shape, not a bigger instance.

The check showed a GIN index already existed on those columns. The planner was not using it on the development database. Poor cardinality and stale statistics made the planner skip it. The fix was a different index shape. A GIN index with jsonb_path_ops fits containment queries on JSON better than the default GIN shape.

The query went from about 5000 ms to about 600 ms. That is more than 8 times faster. Staging comparisons showed no mismatches, so it went to production cleanly.

Sourcing note: this one comes from a pganalyze case study, not from Notion's own blog. It quotes named Notion engineers. I keep it because the lesson is checkable. The index existed, the planner ignored it, and the fix was the index shape, not a bigger instance. When a query is slow, read the plan before you buy hardware.

PostgreSQL

Adyen: one partition hit 30 TB, and the index rebuild took days

30 TBBiggest single partition. An index rebuild there takes days.
15 minTotal downtime to upgrade a 50 TB cluster by snapshot.
5,000+ TPSTransactions per second at the 2018 upgrade.
20xFaster queries after aligning partition boundaries across tables.

The problem

Biggest partition over 30 TB. Index rebuild takes days. xmin stays blocked, bloat grows, wraparound ticks. A normal upgrade on 50 TB would take 4 to 5 weeks of downtime, rough calculation. Payments cannot go down for a month. Each new partition added about 10 percent CPU. HOT updates stopped on one table. Its WAL doubled, about half the whole cluster's WAL.

The 15 minute upgrade

apppayments queue
→
primaryupgrade, 3 to 5 min
→
snapshotvolumes, up to 10 min
→
replica 1online and current
→
replica 2online and current
Primary upgrade
3 to 5 min
Snapshot volumes
up to 10 min
Copy to replicas
2 to 5 min

Postgres itself took minutes. The infrastructure around it took the rest. One partition over 30 TB, and its index rebuild takes days.

The solution

Index per partition first, parent index last. Only way it finishes before wraparound risk grows. Upgrade: stop traffic, queue payments. Primary in 3 to 5 minutes. Snapshot volumes in up to 10 minutes. Copy snapshot to replicas in 2 to 5 minutes. Reopen only when at least two replicas are online and current. Total: 15 minutes. 2018 scale: 5,000 plus transactions per second. Under 10 TB in 2015, 50 TB at upgrade, biggest cluster 74 TB. One primary, at least three replicas across data centers. 768 GB RAM per server, 150 plus TB raw SSD per server. CPU climb: prepared statements, custom plans. Custom plan saved a few milliseconds in execution, cost about 100 milliseconds in prepare time. Forcing generic plans fixed it. Fillfactor: 95 percent some tables, 70 percent others. Test per table, watch WAL. Aligned partition boundaries made some queries 20 times faster.

Let's talk about the design

The upgrade had two shapes. Adyen picked the snapshot method because payments cannot stop. Indexes went per partition first because one big index would never finish before wraparound risk grew. Generic plans won because prepare time cost more than execution time saved.

Cost: the snapshot method needs traffic stopped and payments queued.
Gives up: per-query optimal plans, in exchange for cheaper prepare time.
Cost: fillfactor has no fixed answer. It is set per table.

The part I would copy is the snapshot method. It is the fastest way to move a database you cannot stop.

Useful if your biggest partition is past 1 TB.

PostgreSQL

Notion again: the DMS copy came back three times the size

The problem

20 TB table, 16 partitions per database, about 1 TB each. Postgres 12 to 15 via DMS. Copy moved about 1 TB in 12 to 14 hours. New tables came back about 3 times the on-disk size. Cause was bloat, not TOAST. Vacuum could not keep up. Investigation took about 2 weeks. DMS work had already taken about 4 months.

The copy tripled

DMS copyabout 1 TB in 12 to 14 hours
→
new tablesabout 3 times the size
→
full vacuumsize back
Before copy, 1 TB per partition
1 TB
After DMS copy
about 3 TB

Bloat, not TOAST. Check bloat before you trust the size after a logical copy.

The solution

A full vacuum brought the size back. That is the whole fix. Vacuum is part of the migration, not cleanup after it. After a logical copy, check bloat before you trust the size.

Let's talk about the design

A logical copy with DMS moves rows, not the physical layout. The new tables land without the source's vacuum history, so bloat on arrival is part of the design, not a surprise. That is why the full vacuum belongs in the migration plan, not after it.

Cost: DMS is simple, but the copy is slow.
Cost: bloat shows up late, at the end of the copy.

The part I take from this is the size check. I would check bloat before trusting the size after any logical copy.

Read this before a version upgrade by logical copy.

PostgreSQL

Figma: they stayed on Postgres and built the sharding layer themselves

The problem

One Postgres instance. In 2020, Figma ran a single Postgres database on AWS's largest physical instance. Stack grew almost 100 times in four years. Some tables hit several terabytes and billions of rows. Vacuum hurt reliability. Biggest write tables neared the RDS IOPS ceiling. A table is the smallest unit of vertical partitioning. That lever was done. The database was not slow. It was out of moves. CockroachDB, TiDB, Spanner, Vitess: good systems. The move meant migrating the most critical data and rebuilding all operational knowledge. NoSQL meant rewriting almost the whole backend. Honest note: the post gives no QPS, latency percentiles, exact shard counts, exact table sizes, first table name, team size, or cost. Those gaps stay open.

DBProxy routing

appSQL query
→
DBProxyparse, plan, route
→
PgBouncerpool
→
one shardshard key present
no shard keyscatter gather
→
shard 1
shard 2
shard 3
shard 4

Logical views first. Rollout by percentage, rollback in seconds.

The solution

DBProxy: Go service between the app and PgBouncer. Parses SQL to an AST, pulls the shard key, routes the query. No key means scatter gather to every shard. Same-key tables live in a colo. Joins and transactions work on that key inside a colo. Keys: UserID, FileID, OrgID. Almost every table fits one. No single key worked for all tables. Logical sharding first: logical shards as Postgres views on the same unsharded database. Rollout by percentage, rollback by config change. Views cost under 10 percent worst case. Then logical replication copy, split for real. Full dataset copied to each shard, reads and writes fenced to that shard's rows. Avoided filtered logical replication. First table: about 9 months. First cutover, September 2023: 10 seconds partial availability on primaries, nothing on replicas, no latency or availability regression after. Shadow planning on live traffic. Queries and plans logged to Snowflake. Query language covering the most common 90 percent. Range scans and point lookups in. Joins only inside one colo and on the shard key. Hash routing, since sequential or time prefixed IDs would pile onto one shard. Hashing hurts range scans on the shard key, accepted as rare. Topology updates reach DBProxy in under 1 second.

Let's talk about the design

Figma stayed on Postgres and built the sharding layer themselves. The engine was not the problem. The move was. Logical views first meant the rollout could go by percentage and roll back with a config change.

Gives up: atomic cross shard transactions. A write can land on one shard and fail on another.
Gives up: foreign keys and global unique indexes across shards.
Cost: schema changes need coordination everywhere.
Gives up: globally unique IDs, a sharding friendly ORM, and one click resharding, all still open at March 2024.

The part I would copy is the logical views first rollout. Prove the routing before moving a row.

Useful if your biggest tables are past a few terabytes and vertical partitioning is done.

PostgreSQL

Notion: sharding fixed the live database, and the warehouse broke next

200B+Blocks by 2024, up from 20 billion in early 2021.
1 week to 2 hoursFull historical resync, after the Hudi pipeline.
More than $1MNet saving in 2022, higher in 2023 and 2024.

The problem

Notion sharded Postgres and the live product got room to breathe. The warehouse did not. Every change still had to move out of the sharded Postgres databases into Snowflake. Fivetran ran out of memory on bursts. Closed source, few knobs. Some hourly syncs took more than an hour and ran into the next sync. Write pattern: only about 1 percent of blocks changed. Edits random, not recent. A full reload for 1 percent of changed rows is the wrong shape. That pointed to Hudi. No query rate or latency numbers in the material. None are given here.

The warehouse after sharding

Postgres shards32 databases
→
DebeziumCDC
→
Kafkachange stream
→
Hudi on S3upserts
→
SparkETL
Fivetranstayed, small data only
→
Snowflakesmall data sets, third party sources
Full resync before
1 week
Full resync after
2 hours

Pick the lake for your upsert pattern, not for the brand name.

The solution

Postgres to Debezium CDC to Kafka to Hudi on S3, then Spark for ETL. Fivetran stayed only for small data sets and third party sources. Blocks: more than 200 billion, up from 20 billion in early 2021. Data at rest: hundreds of terabytes, even compressed. Full resync: 1 week to 2 hours. Incremental syncs every 4 hours. Net saving: over a million dollars in 2022, and more in 2023 and 2024. The lake then fed Notion AI Q and A: an offline embedding batch, run once per workspace to bootstrap the vector database, plus an online Kafka catchup within a day.

Let's talk about the design

The choice followed the write pattern. Few rows changed, and the edits were random. That is the shape Hudi is built for. The tuning follows the same pattern: files sized to limit write amplification, a sort key that clusters recent changes, and partitioning that fits the sharded Postgres layout they already had.

Cost: a pipeline to run: Debezium, Kafka, Hudi, Spark.
Cost: Fivetran stays for small data sets, so two paths to maintain.
Gives up: real time syncs. Incremental syncs only.

The part I would copy is the pattern check first. Few changed rows, mostly updates, random edits. That is what picks the lake.

Read this when your warehouse syncs start missing their window.

PostgreSQL

GitLab: five backups, none of them worked

300 GBProduction data deleted in seconds. About 4.5 GB left.
18 hoursDowntime while they rebuilt from a staging copy.
6 hoursOf writes lost for good: 5,000 projects, 5,000 comments, about 700 users.

The problem

On January 31, 2017, spam accounts and a background job that was deleting an employee's data loaded GitLab's Postgres primary. The secondary fell so far behind that it could no longer catch up. GitLab was not running WAL archiving, so the old WAL segments the secondary still needed were already gone. The only fix was to wipe the secondary and copy the whole database again with pg_basebackup.

Each safety net failed in its own quiet way

✗pg_dump to S3, every 24 hours: the bucket was empty
✗Azure disk snapshots: on the file server, not the database servers
✗the replica: wiped an hour earlier for the re-sync
✗WAL archiving: never turned on
staging copy, 6 hours oldthe one that worked, and it was never meant to be a backup

Late at night, tired, the engineer ran the wipe in the wrong terminal. The command was meant for db2, the replica. It ran on db1, the primary. The two hostnames differ by one character.

The solution

He stopped the command within seconds, but of around 300 GB only about 4.5 GB was left. Then they went to the backups. The pg_dump job had been running a 9.2 client against a 9.6 server, so it errored out and made nothing. Its failure emails bounced, because DMARC was not enabled for them. The disk snapshots covered the file server, not the database. The replica was already wiped. The copy that saved them was a staging copy taken about six hours earlier for a load test. They copied it back, rebuilt Postgres, and brought GitLab.com up against it. The site was down about 18 hours, and about six hours of writes were lost for good. They live-streamed the whole recovery.

Their postmortem, published February 10, 2017, put it in one line: out of five backup and replication techniques deployed, none were working reliably or set up in the first place. The root cause was not the wrong terminal. It was that nobody was responsible for testing the restore.

Let's talk about the design

The setup had five safety nets and no owner. WAL archiving was off, so a lagging replica could only be rebuilt from scratch. The backup job failed silently for years, and the alert emails bounced. Each mechanism failed in a different quiet way, so no single check would have caught all of them. The fix was not a sixth backup. It was an owner and a restore test.

Cost: five safety nets to maintain, and no one owned testing any of them.
Gives up: WAL archiving was off, so a lagging replica could only be rebuilt by wiping it.
Gives up: the only good copy was six hours old, so recovery cost six hours of writes.

The part I would copy is the restore test. A backup you have never restored is a guess.

Read this before you trust a backup you have never restored.

PostgreSQL

Instacart: they split the database by domain, not by shard key

6 TBRDS limit they were about to hit, raised to 12 TB before they finished.
A few dozenTables needed by more than one domain, out of hundreds.
2 databasesCustomers and Fulfillment, one app talking to both.

The problem

Instacart ran its marketplace on two databases, a primary and an item catalog, and they were running out of space. The RDS limit was 6 TB, raised to 12 TB before they finished. Replicas already handled most of the reads, and the write load was manageable. Space was the urgent problem, and isolation was the other one. Instacart is a four-sided marketplace: customers, shoppers, retailers, and consumer packaged goods companies. If the fulfillment systems went down, shoppers could not work, and the company could not deliver orders.

One database, two domains, then two databases

Before: one primary

Hundreds of tables. Four domains reading and writing the same tables. One outage hits everyone.

After: two databases

Customers and Fulfillment. Each domain owns its tables. Shared tables are copied one way.
owning domain writesone writer per table
→
data pump publishes the changeafter the write commits
→
copied table on the other sideread-only, checked by a sweeper

Only a few dozen tables crossed domains. That is what made the domain split possible.

The solution

They examined their hundreds of tables and found only a few dozen were needed by more than one domain. They created a dedicated Fulfillment database to hold everything needed to complete an order. For each shared table they picked one owning domain, where the table and its writes live. Other domains got a copied table, read-only at the Postgres user level and in the app. They tried foreign data wrappers with materialized views first. The experiment was not a failure, but refreshing the views looked problematic at scale. So they built an application-level data pump: after a write commits, the model publishes the change over their pub/sub, and a consumer applies it to the copied table.

Moving a table took a fixed sequence: consolidate all writes into the owning domain, build the model in the new database, backfill the data with their pgsync tool, dual-write to both databases, then cut over in a single pull request. A sweeper job re-checks every copied model on a schedule, comparing rows by updated_at and id. Before cutover they compared both sides with a simple count and average id check on rows older than ten seconds.

Let's talk about the design

The split followed the domain boundary, not a shard key, because only a few dozen tables crossed domains. Sharding by key would have split tables that belonged together. Microservices would have meant the same split with more services to run. The data pump won over foreign data wrappers because the app controls exactly which changes each domain receives, and a domain can ignore updates it does not care about.

Cost: eventual consistency. The copied table lags the master, so a read can be stale.
Cost: a sweeper to run forever, re-checking every copied model for drift.
Gives up: joins across the two databases. A query can no longer join a customers table to a fulfillment table in one place.
Cost: every table move is a project: consolidate writes, backfill, dual-write, cut over.

The part I would copy is the owning domain rule. One writer per table makes the copy simple, and the copy is the whole design.

Read this when one database holds two domains and you are running out of room.

MySQL · five stories

MySQL: failover, upgrades, and the shard key

GitHub shows the two halves of operating MySQL at scale: the failover you automate, and the upgrade you survive. Slack and Shopify show the same lesson from the other side: the shard key decides everything, and the migration only works if you can compare both copies before you trust one. Dropbox shows what happens when one tenant outgrows a shard entirely.

MySQL

Dropbox: one shard outgrew one machine, so they built a new store

The problem

Metadata lived in sharded MySQL: thousands of servers, tens of millions of queries per second at single digit millisecond latency. Then one shard was going to outgrow a single machine. The shard could not be split, and the machine could not grow. Expanding meant splitting every machine in two, doubling the fleet for a fraction more data.

One tenant outgrew one shard

Before: sharded MySQL

One shard is the outlier, outgrowing its machine. The rest sit idle.

After: Panda ranges

range 1
range 2
range 3
range 4

One shard split into many ranges. Capacity grows a range at a time.

The solution

Panda: a distributed key value store, built in house. The key space splits many times into smaller ranges. Sits between Edgestore and sharded MySQL. ACID transactions, including atomic cross shard transactions through two phase commit. Incremental capacity growth and rebalancing.

Let's talk about the design

No sharded design can split one shard, so the fix had to be a store that splits ranges. Dropbox built Panda in house instead of trying to stretch MySQL further.

Cost: a second store to run, forever.
Cost: two phase commit across shards, for atomic cross shard writes.

The part I would copy is splitting by range. That is what kept the move safe.

Read this when one shard can outgrow a machine.

MySQL

GitHub: the master died, and nobody woke up

GitHub stores its metadata in MySQL. Issues, pull requests, comments, organizations, notifications. One master takes the writes. Replicas serve the reads. A dead master used to mean a human waking up and running a failover by hand. That was too slow for the availability they wanted.

One master, replicas, one watcher

mastertakes writes
Orchestratorwatches the cluster, asks the replicas what happened
guardrailone automatic failover per cluster, then blocked 30 minutes or until a human says go
replica 1serves reads
replica 2serves reads
a replica, promotedmaster in 30 seconds or less

Detection plus a flap guard, not just a tool. Automation without a guardrail is a second outage waiting to happen.

They run Orchestrator. It watches the replication topology and runs the failover itself. Their target was detection plus recovery in 30 seconds or less, with minimal loss of hosts. Detection is holistic. It asks the replication cluster what happened instead of trusting one monitor. They tested its decisions for months before they turned on automatic recovery.

The guardrails matter as much as the tool. Flapping is not allowed. Orchestrator runs at most one automatic failover per cluster in a set time period. After a failover, the cluster is blocked from another one for 30 minutes, or until a human acknowledges it. It also handles a dead intermediate master, so a middle layer failing does not strand the replicas below it. Later work added Pseudo-GTID handling for replicas that fall behind or go unreachable.

My take: the failover tool is only half the story. The other half is the policy around it. One failover, then a block, then a human. Automation without a flap guard is a second outage waiting to happen. I check for that guardrail in any failover tool before I trust it.

MySQL

GitHub: upgrading 1,200 MySQL hosts without dropping a query

The fleet was 1,200-plus MySQL hosts, a mix of Azure VMs and bare metal. They held 300-plus TB of data and served 5.5 million queries per second across 50-plus clusters. MySQL 5.7 was near end of life, so they moved to 8.0. Planning, testing, and the upgrade took over a year.

1,200+ hostsAzure VMs and bare metal, 300+ TB of data, 5.5M queries per second, 50+ clusters.
1+ yearPlanning, testing, and the upgrade. Prep started July 2022.
24 hoursValidation cycle per cluster before the rollback path closed.

The upgrade, in four steps

1

Jul 2022: 8.0 added to CI. 5.7 and 8.0 side by side, so regressions showed up before production.

2

Dual chains. 8.0 replicas online, one data center at a time, behind 5.7 primaries kept as rollback.

3

Promote. An 8.0 replica becomes primary through Orchestrator. No primary was ever upgraded in place.

4

24-hour validation. Then the 5.7 rollback path closes. The collation change, roles, and one replica bug bit along the way.

Prep started in July 2022. They added 8.0 to CI and ran 5.7 and 8.0 side by side, so regressions showed up before production. They never upgraded a primary in place. They brought 8.0 replicas online, one data center at a time, and promoted an 8.0 replica to primary through Orchestrator.

The rollback path was the design, not an afterthought. They kept 5.7 standbys behind the 8.0 primaries, so they could roll back without a disruption. Each cluster got a 24-hour validation cycle before the rollback path closed. The collation change bit them. Version 8.0 defaults to utf8mb4_0900_ai_ci, and 5.7 does not understand it, so backward replication broke. Roles broke replication too. A replica_preserve_commit_order bug, fixed in 8.0.28, was another one. A WHERE IN list with tens of thousands of values crashed MySQL.

My take: the upgrade was a replication project, not a version bump. If you cannot roll back, you cannot upgrade a fleet this size. I would copy the shape. Dual chains, one cluster at a time, a fixed validation window, and the rollback kept alive until the window closes.

MySQL

Slack: the shard key was the hot spot

Slack started moving its MySQL datastores to Vitess in 2017. By December 2020, 99 percent of all MySQL traffic ran on Vitess. Peak load was 2.3 million queries per second: 2 million reads, 300 thousand writes. Median latency was 2 ms. p99 was 11 ms.

2.3M QPSPeak load. 2M reads, 300K writes. Median 2 ms, p99 11 ms.
99%Of all MySQL traffic on Vitess by December 2020.
+50% in a weekQuery rates in March 2020. They split a busy keyspace live and kept serving.

Workspace sharding vs channel ID resharding

Before: workspace sharding

hot
idle
idle
idle

One big customer lands on one shard. It runs hot while the rest sit idle.

After: channel ID resharding

Traffic spreads. In March 2020 query rates rose 50 percent in one week. They split a busy keyspace live and kept serving.

write to bothdouble-write backfill
→
read from bothdouble-read diffing
→
cut overonly after both copies agree

Sharding is a routing problem before it is a capacity problem.

The old shape was workspace sharding on active-active MySQL. It worked until a big customer landed on one shard and made it hot while the rest sat idle. Enterprise Grid and Slack Connect broke the model further, because one user's data now spanned workspaces.

The move was a re-shard by channel ID under Vitess. The migration ran on double-write backfill plus double-read diffing. Write to both, read from both, compare, and only then cut over. In March 2020, query rates rose 50 percent in one week. They split a busy keyspace horizontally and kept serving. Without resharding, they wrote, their largest customers would have had downtime.

The extra win came later. The same Vitess clusters now run their International Data Residency product in six regions. My take: sharding is a routing problem before it is a capacity problem. The channel key fixed the hot spot that the workspace key created. Pick the key from the access pattern, and build the migration so you can compare both copies before you trust one.

MySQL

Shopify: the verifiers were the project

The Shop app backend is Ruby on Rails on a single MySQL, their managed system called KateSQL. Growth was steep. Disk grew to many terabytes. Schema migrations took weeks. When the database got busy, they throttled background jobs.

The five verifiers, checked before any data moved

✓every query carries the sharding key
✓no transaction crosses databases
✓no transaction crosses shards
✓no write crosses shards
✓no join crosses keyspaces
step 1insert the row in the new shard
→
step 2delete it from the old shard

A commit that lands on one shard and fails on another does not roll back the first. Build the check first, move the data second.

The first fix was federation. GhostFerry moved whole tables onto new databases, and new domains got their own databases. That bought time and limited blast radius. It hit its own wall. Migrations on big tables still took weeks, joins across databases got hard, and the app had to know where every table lived.

The second fix was Vitess, sharded by user_id. User-owned data went into a sharded users keyspace. Everything else went into an unsharded global keyspace. Before moving, they built query verifiers into the app layer. The verifiers caught queries missing the sharding key, transactions crossing databases, transactions crossing shards, writes crossing shards, and queries joining across keyspaces. They removed cross-shard transactions entirely. A commit that lands on one shard and fails on another does not roll back the first. Moving a row across shards became two steps: insert in the new shard, then delete from the old one.

My take: the verifiers were the project. Shopify wrote that if they could credit one thing for the move working, it was the correctness of the verifiers. Sharding without a check that every query carries its key is a data corruption incident on a schedule. Build the check first, move the data second.

MongoDB · one story

MongoDB: the cache is the database

One story so far, and it is the classic MongoDB failure: the working set outgrows RAM, the cache stalls, and the connection storm finishes the job. The fix is timeouts, throttling, and reads off the primary, not a bigger instance.

MongoDB

Fortnite: a release changed the call pattern, and the cache ran out

20:44 to 14:30 UTCAccount service down, April 11 to 12, 2018.
7 attemptsTo bring services back online over the next 17 hours.
4.5xPage evictions on the primary above baseline before issues started.
29 GBDirty cache on the primary when mongod started having communication issues.

The problem

On April 11, 2018, Fortnite shipped version 3.5, and logins stopped working for all players. The MongoDB database behind the account service had degraded until the account service was unavailable, from 20:44 UTC on April 11 to 14:30 UTC on April 12. A change in call patterns in 3.5 slowly pushed cache reads up over hours. The dataset no longer fit in RAM, so pages were evicted and read back from disk faster and faster, until evictions ran 4.5 times above baseline. Dirty cache, data written to RAM and not yet to disk, built up to about 29 GB, and the primary could no longer keep up with both request load and replication load. The cache pressure was enough to overload the primary and trigger leader elections.

The cascade: slow queries became a connection storm

3.5 call patterncache reads climb for hours
→
cache pressureevictions 4.5x baseline
→
slow queriesclients open new connections
→
connection stormmax connections hit, nodes drop
Page evictions, baseline
1x
Page evictions, at the tipping point
4.5x

Short timeouts turned slow queries into new connections. New connections turned a slow database into an unreachable one.

The solution

They tried to bring services back seven times over 17 hours, and each attempt ran into the same cascade. The fixes that stuck: the socket timeout went from 200 ms to 30 seconds, so the app stopped closing connections at the first slow query and churning new ones. Queries got a 500 ms maxTimeMS instead, so the database kills a slow operation and keeps the connection open. The app moved behind mongos, the shard router, which has more connection controls than the driver, plus a throttle on opening new connections to the database, about 50 at a time. A code path that pinned specific reads to the primary was changed to read from secondaries.

The Friend Suggestions bug, re-querying the list on every friends list refresh, had pushed one API call to 300 times its normal load, and the hotfix tunables meant to control it were broken. Version 3.5.1 fixed both. Ephemeral data, short-lived single-use OAuth authorization codes, moved out of MongoDB into Redis. Better caching of account data cut database lookups at least in half. Recovery meant removing traffic and adding clients back a little at a time from a cold start. Two replica set members were never explained: their CPU hit 100 percent, almost all of it in kernel time, logging stopped, and the process looked hung. Those two were isolated and replaced.

Let's talk about the design

The storm was the killer, not the data size. A bigger instance would have held more cache, but the connection churn would still have knocked nodes offline once the database slowed. Throttling connections at mongos protected the replica set's own communication, which is what kept members from dropping out.

Cost: mongos in the path, another hop between the app and the database.
Cost: a 30 second socket timeout means a slow query holds its connection longer before the client gives up.
Gives up: reads moved to secondaries can be stale, so anything that needs the freshest account data stays on the primary.

The part I would copy is the connection throttle. When the database slows down, the client must not be allowed to make it worse.

Read this when your timeouts are short and your connections spike under load.

Before you leave, check this: pick the story closest to your own system, and name the one decision in it you have not made yet. The shard key, the failover guardrail, the rollback path, the index shape. That decision is the incident you have not had yet. The hosting side of the same question, on-prem, EC2, Kubernetes, or managed, is on the Cloud page.