Every page on this site explains a mechanism. This page shows what happens when the mechanism meets real traffic. Sixteen stories, all from companies that wrote up what broke. Each one has the numbers, the fix, and the lesson I took from it. The source is linked at the end of each story.
None of these are my incidents. I read them the way I read a postmortem: what broke first, what the numbers were, and what decision would have prevented it.
Most of these are the same failure wearing different clothes: a physical layout decision, made early, priced later at scale. Adyen and Figma show it at the far end, where the partition and the shard key decide everything before the first query runs. One is a planner that had the right index and refused to use it. GitLab and Instacart are the exceptions: a backup nobody tested, and a split by domain instead of by shard key.
Uber started on Postgres. One monolithic backend, one database holding the trip data. As the company grew, the database became the constraint. The post names five limitations: inefficient writes, inefficient replication, table corruption, poor replica MVCC, and hard upgrades. The first two are the expensive ones, and they share one root cause.
One logical update, four physical writes
A table with a dozen indexes rewrites all twelve on every update. The cost is physical, not logical.
They were running Postgres 9.2. Every update wrote a brand-new row version, and every secondary index pointed at the row's physical spot on disk. One logical update became 4 or more physical writes. A table with a dozen indexes rewrote all twelve on every update. The WAL stream was physical too, so every extra byte crossed the network to replicas in the other data center. A bug in 9.2 returned duplicate rows on replicas after a timeline switch, and the only fix was a full resync. Long queries on replicas got killed by cleanup. Upgrades needed full downtime, so old instances stayed on 9.2 while 9.5 was current.
The fix was Schemaless, built on MySQL and InnoDB. In InnoDB, a secondary index points at the primary key, not at a disk address. An update touches the row in place and leaves the other indexes alone. Replication is logical, through the binlog, so the stream across data centers is far smaller.
Write amplification went away. Replication got cheaper. Upgrades stopped needing full downtime. The lesson I take is not that Postgres is slow. The lesson is that index layout and replication format are the whole cost at write-heavy scale. One more thing: the post is from July 2016, about Postgres 9.2. Later versions fixed some of the named bugs. The layout difference is still real.
Source: Why Uber Engineering Switched from Postgres to MySQL, Uber Engineering Blog, July 26, 2016.
Notion ran its product on one Postgres monolith. Every paragraph, heading, image, and to-do is a block, and every block is a row in one giant table. By mid-2020, usage had outgrown the single database.
Before and after: one monolith, 480 shards
32 physical databases, 15 logical shards each. The shard key is the whole project.
The trigger was specific. VACUUM started stalling, so dead tuples stopped being reclaimed and disk space stopped coming back. Behind that sat transaction ID wraparound. Postgres stops all writes rather than overwrite old data. That is not a slowdown. That is the product stopping.
The design was application-level sharding, routing by workspace ID so one workspace's data stays together. They built 480 logical shards spread across 32 physical databases. They set their own caps: 500 GB per table, 10 TB per physical database, 60K IOPS. Logical replication could not keep up with the block table's write volume during the first snapshot, so they moved to an audit-log double-write. The backfill ran 3 days on an m5.24xlarge with 96 CPUs. They used dark reads to verify, and the verification code was written by different people than the migration code. The cutover took 5 minutes of scheduled downtime.
Their own lessons: shard earlier, aim for zero downtime, and use a combined primary key instead of a separate partition key. Mine: the shard key decision is the whole project. Workspace ID kept queries local. Pick the key from the query pattern, and pick it before you are forced to.
Two details from the same post. They prepared and tested a reverse audit log, so they could replay sharded writes back to the monolith if they had to roll back. They never needed it. They picked 480 shards because the number has many factors, so they could grow from 32 to 40 to 48 hosts and keep shards even; with 512 they would have had to double from 32 to 64. They used separate tables per schema, schema001.block and so on, instead of one natively partitioned table per database, so all routing stays in the application.
Source: Herding elephants: Lessons learned from sharding Postgres at Notion, Notion Blog, October 6, 2021.
Part 2 of the Notion sharding story above. The 5 minute cutover and the 3 day backfill belong to the first sharding, not this one.
32 databases running hot. Some shards passed 90 percent CPU at peak. Many near full on provisioned disk bandwidth. PgBouncer hitting connection limits. A new year spike was coming. Out of headroom. The database was not broken. It was full.
32 to 96, and the pool split first
When you reshard, reshard the pool too. The connection math is what bites you.
No bigger machines. Tripled the fleet, 32 to 96, same 480 logical shards, 15 schemas per database down to 5. Smaller instances and disks to hold cost down. Disk was not the bottleneck. Copy with Postgres logical replication: 3 publications per old database, 5 schemas per publication. Indexes skipped during the copy, rebuilt after. Sync time fell from 3 days to 12 hours. The pool nearly broke the plan. About 100 PgBouncer instances, up to 6 connections per shard, 600 per shard. A naive cutover would push that to 18 per old shard. Cutting connections made queries back up in testing. So PgBouncer was sharded first, into 4 groups with 24 databases each. That also limited a PgBouncer failure to 25 percent of the fleet. Failover: pause traffic in PgBouncer, let server connections finish, check replication caught up, flip the mapping, reload PgBouncer, revoke the old login, reverse replication for rollback. Checkpoints and rollback at every step. Full process in staging first, tested multiple times. Dark reads checked the copy first: parallel reads to the follower, queries returning up to 5 rows, small sample, 1 second pause for replication to catch up. Near 100 percent equivalence. Outcome: no observed user downtime. Worst case about 1 second of saving spinner. After the move, CPU and IOPS sat around 20 percent at peak.
Notion picked tripling over bigger machines because bigger machines would hit the same wall later, and one big box cannot be tuned per shard. PgBouncer got sharded first because the connection math was the real limit.
The part I would copy is sharding the pool first. The connection math is what bites you.
Read this when your fleet is hot and out of headroom.
Source: The Great Re-shard: adding Postgres capacity (again) with zero downtime, Notion Blog, July 17, 2023.
Same company, a different problem. Notion ran one large Postgres database on RDS from 2015 to 2020. A new feature was already 20 percent of the application's query load before it shipped to everyone. One of its queries filtered JSON on space_id and permission.
Same query, two index shapes
More than 8 times faster. The index existed. The planner ignored it. The fix was the index shape, not a bigger instance.
The check showed a GIN index already existed on those columns. The planner was not using it on the development database. Poor cardinality and stale statistics made the planner skip it. The fix was a different index shape. A GIN index with jsonb_path_ops fits containment queries on JSON better than the default GIN shape.
The query went from about 5000 ms to about 600 ms. That is more than 8 times faster. Staging comparisons showed no mismatches, so it went to production cleanly.
Sourcing note: this one comes from a pganalyze case study, not from Notion's own blog. It quotes named Notion engineers. I keep it because the lesson is checkable. The index existed, the planner ignored it, and the fix was the index shape, not a bigger instance. When a query is slow, read the plan before you buy hardware.
Source: How Notion Runs PostgreSQL at Scale on Amazon RDS with pganalyze, pganalyze, February 27, 2025.
Biggest partition over 30 TB. Index rebuild takes days. xmin stays blocked, bloat grows, wraparound ticks. A normal upgrade on 50 TB would take 4 to 5 weeks of downtime, rough calculation. Payments cannot go down for a month. Each new partition added about 10 percent CPU. HOT updates stopped on one table. Its WAL doubled, about half the whole cluster's WAL.
The 15 minute upgrade
Postgres itself took minutes. The infrastructure around it took the rest. One partition over 30 TB, and its index rebuild takes days.
Index per partition first, parent index last. Only way it finishes before wraparound risk grows. Upgrade: stop traffic, queue payments. Primary in 3 to 5 minutes. Snapshot volumes in up to 10 minutes. Copy snapshot to replicas in 2 to 5 minutes. Reopen only when at least two replicas are online and current. Total: 15 minutes. 2018 scale: 5,000 plus transactions per second. Under 10 TB in 2015, 50 TB at upgrade, biggest cluster 74 TB. One primary, at least three replicas across data centers. 768 GB RAM per server, 150 plus TB raw SSD per server. CPU climb: prepared statements, custom plans. Custom plan saved a few milliseconds in execution, cost about 100 milliseconds in prepare time. Forcing generic plans fixed it. Fillfactor: 95 percent some tables, 70 percent others. Test per table, watch WAL. Aligned partition boundaries made some queries 20 times faster.
The upgrade had two shapes. Adyen picked the snapshot method because payments cannot stop. Indexes went per partition first because one big index would never finish before wraparound risk grew. Generic plans won because prepare time cost more than execution time saved.
The part I would copy is the snapshot method. It is the fastest way to move a database you cannot stop.
Useful if your biggest partition is past 1 TB.
Source: Updating a 50 terabyte PostgreSQL database, Adyen, March 14, 2018. Companion: To 100TB, and beyond!, Postgres.FM 100, June 7, 2024 (full transcript).
20 TB table, 16 partitions per database, about 1 TB each. Postgres 12 to 15 via DMS. Copy moved about 1 TB in 12 to 14 hours. New tables came back about 3 times the on-disk size. Cause was bloat, not TOAST. Vacuum could not keep up. Investigation took about 2 weeks. DMS work had already taken about 4 months.
The copy tripled
Bloat, not TOAST. Check bloat before you trust the size after a logical copy.
A full vacuum brought the size back. That is the whole fix. Vacuum is part of the migration, not cleanup after it. After a logical copy, check bloat before you trust the size.
A logical copy with DMS moves rows, not the physical layout. The new tables land without the source's vacuum history, so bloat on arrival is part of the design, not a surprise. That is why the full vacuum belongs in the migration plan, not after it.
The part I take from this is the size check. I would check bloat before trusting the size after any logical copy.
Read this before a version upgrade by logical copy.
Source: To 100TB, and beyond!, Postgres.FM 100, June 7, 2024 (full transcript).
One Postgres instance. In 2020, Figma ran a single Postgres database on AWS's largest physical instance. Stack grew almost 100 times in four years. Some tables hit several terabytes and billions of rows. Vacuum hurt reliability. Biggest write tables neared the RDS IOPS ceiling. A table is the smallest unit of vertical partitioning. That lever was done. The database was not slow. It was out of moves. CockroachDB, TiDB, Spanner, Vitess: good systems. The move meant migrating the most critical data and rebuilding all operational knowledge. NoSQL meant rewriting almost the whole backend. Honest note: the post gives no QPS, latency percentiles, exact shard counts, exact table sizes, first table name, team size, or cost. Those gaps stay open.
DBProxy routing
Logical views first. Rollout by percentage, rollback in seconds.
DBProxy: Go service between the app and PgBouncer. Parses SQL to an AST, pulls the shard key, routes the query. No key means scatter gather to every shard. Same-key tables live in a colo. Joins and transactions work on that key inside a colo. Keys: UserID, FileID, OrgID. Almost every table fits one. No single key worked for all tables. Logical sharding first: logical shards as Postgres views on the same unsharded database. Rollout by percentage, rollback by config change. Views cost under 10 percent worst case. Then logical replication copy, split for real. Full dataset copied to each shard, reads and writes fenced to that shard's rows. Avoided filtered logical replication. First table: about 9 months. First cutover, September 2023: 10 seconds partial availability on primaries, nothing on replicas, no latency or availability regression after. Shadow planning on live traffic. Queries and plans logged to Snowflake. Query language covering the most common 90 percent. Range scans and point lookups in. Joins only inside one colo and on the shard key. Hash routing, since sequential or time prefixed IDs would pile onto one shard. Hashing hurts range scans on the shard key, accepted as rare. Topology updates reach DBProxy in under 1 second.
Figma stayed on Postgres and built the sharding layer themselves. The engine was not the problem. The move was. Logical views first meant the rollout could go by percentage and roll back with a config change.
The part I would copy is the logical views first rollout. Prove the routing before moving a row.
Useful if your biggest tables are past a few terabytes and vertical partitioning is done.
Source: How Figma's Databases Team Lived to Tell the Scale, Figma Blog, Sammy Steele, March 14, 2024. Companion: How Figma built DBProxy for sharding Postgres, pganalyze, 5mins of Postgres E106 (transcript). Companion: 5 Lessons from 5 Years of Building Databases at Scale, Sammy Steele, Systems Distributed '24.
Notion sharded Postgres and the live product got room to breathe. The warehouse did not. Every change still had to move out of the sharded Postgres databases into Snowflake. Fivetran ran out of memory on bursts. Closed source, few knobs. Some hourly syncs took more than an hour and ran into the next sync. Write pattern: only about 1 percent of blocks changed. Edits random, not recent. A full reload for 1 percent of changed rows is the wrong shape. That pointed to Hudi. No query rate or latency numbers in the material. None are given here.
The warehouse after sharding
Pick the lake for your upsert pattern, not for the brand name.
Postgres to Debezium CDC to Kafka to Hudi on S3, then Spark for ETL. Fivetran stayed only for small data sets and third party sources. Blocks: more than 200 billion, up from 20 billion in early 2021. Data at rest: hundreds of terabytes, even compressed. Full resync: 1 week to 2 hours. Incremental syncs every 4 hours. Net saving: over a million dollars in 2022, and more in 2023 and 2024. The lake then fed Notion AI Q and A: an offline embedding batch, run once per workspace to bootstrap the vector database, plus an online Kafka catchup within a day.
The choice followed the write pattern. Few rows changed, and the edits were random. That is the shape Hudi is built for. The tuning follows the same pattern: files sized to limit write amplification, a sort key that clusters recent changes, and partitioning that fits the sharded Postgres layout they already had.
The part I would copy is the pattern check first. Few changed rows, mostly updates, random edits. That is what picks the lake.
Read this when your warehouse syncs start missing their window.
Source: Building and scaling Notion's data lake, Notion Blog, July 1, 2024. Companion: Notion's Journey Through Different Stages of Data Scale, Onehouse. Talk: A Hudi Live Event: Notion's journey through different stages of data scale, Thomas Chow and Nathan Louie, Notion Data Platform, December 13, 2023.
On January 31, 2017, spam accounts and a background job that was deleting an employee's data loaded GitLab's Postgres primary. The secondary fell so far behind that it could no longer catch up. GitLab was not running WAL archiving, so the old WAL segments the secondary still needed were already gone. The only fix was to wipe the secondary and copy the whole database again with pg_basebackup.
Each safety net failed in its own quiet way
Late at night, tired, the engineer ran the wipe in the wrong terminal. The command was meant for db2, the replica. It ran on db1, the primary. The two hostnames differ by one character.
He stopped the command within seconds, but of around 300 GB only about 4.5 GB was left. Then they went to the backups. The pg_dump job had been running a 9.2 client against a 9.6 server, so it errored out and made nothing. Its failure emails bounced, because DMARC was not enabled for them. The disk snapshots covered the file server, not the database. The replica was already wiped. The copy that saved them was a staging copy taken about six hours earlier for a load test. They copied it back, rebuilt Postgres, and brought GitLab.com up against it. The site was down about 18 hours, and about six hours of writes were lost for good. They live-streamed the whole recovery.
Their postmortem, published February 10, 2017, put it in one line: out of five backup and replication techniques deployed, none were working reliably or set up in the first place. The root cause was not the wrong terminal. It was that nobody was responsible for testing the restore.
The setup had five safety nets and no owner. WAL archiving was off, so a lagging replica could only be rebuilt from scratch. The backup job failed silently for years, and the alert emails bounced. Each mechanism failed in a different quiet way, so no single check would have caught all of them. The fix was not a sixth backup. It was an owner and a restore test.
The part I would copy is the restore test. A backup you have never restored is a guess.
Read this before you trust a backup you have never restored.
Source: Postmortem of database outage of January 31, GitLab Blog, February 10, 2017.
Instacart ran its marketplace on two databases, a primary and an item catalog, and they were running out of space. The RDS limit was 6 TB, raised to 12 TB before they finished. Replicas already handled most of the reads, and the write load was manageable. Space was the urgent problem, and isolation was the other one. Instacart is a four-sided marketplace: customers, shoppers, retailers, and consumer packaged goods companies. If the fulfillment systems went down, shoppers could not work, and the company could not deliver orders.
One database, two domains, then two databases
Only a few dozen tables crossed domains. That is what made the domain split possible.
They examined their hundreds of tables and found only a few dozen were needed by more than one domain. They created a dedicated Fulfillment database to hold everything needed to complete an order. For each shared table they picked one owning domain, where the table and its writes live. Other domains got a copied table, read-only at the Postgres user level and in the app. They tried foreign data wrappers with materialized views first. The experiment was not a failure, but refreshing the views looked problematic at scale. So they built an application-level data pump: after a write commits, the model publishes the change over their pub/sub, and a consumer applies it to the copied table.
Moving a table took a fixed sequence: consolidate all writes into the owning domain, build the model in the new database, backfill the data with their pgsync tool, dual-write to both databases, then cut over in a single pull request. A sweeper job re-checks every copied model on a schedule, comparing rows by updated_at and id. Before cutover they compared both sides with a simple count and average id check on rows older than ten seconds.
The split followed the domain boundary, not a shard key, because only a few dozen tables crossed domains. Sharding by key would have split tables that belonged together. Microservices would have meant the same split with more services to run. The data pump won over foreign data wrappers because the app controls exactly which changes each domain receives, and a domain can ignore updates it does not care about.
The part I would copy is the owning domain rule. One writer per table makes the copy simple, and the copy is the whole design.
Read this when one database holds two domains and you are running out of room.
Source: Scaling at Instacart: Distributing Data Across Multiple Postgres Databases with Rails, Instacart Engineering, 2018.
GitHub shows the two halves of operating MySQL at scale: the failover you automate, and the upgrade you survive. Slack and Shopify show the same lesson from the other side: the shard key decides everything, and the migration only works if you can compare both copies before you trust one. Dropbox shows what happens when one tenant outgrows a shard entirely.
Metadata lived in sharded MySQL: thousands of servers, tens of millions of queries per second at single digit millisecond latency. Then one shard was going to outgrow a single machine. The shard could not be split, and the machine could not grow. Expanding meant splitting every machine in two, doubling the fleet for a fraction more data.
One tenant outgrew one shard
One shard is the outlier, outgrowing its machine. The rest sit idle.
One shard split into many ranges. Capacity grows a range at a time.
Panda: a distributed key value store, built in house. The key space splits many times into smaller ranges. Sits between Edgestore and sharded MySQL. ACID transactions, including atomic cross shard transactions through two phase commit. Incremental capacity growth and rebalancing.
No sharded design can split one shard, so the fix had to be a store that splits ranges. Dropbox built Panda in house instead of trying to stretch MySQL further.
The part I would copy is splitting by range. That is what kept the move safe.
Read this when one shard can outgrow a machine.
Source: Future-proofing our metadata stack with Panda, a scalable key-value store, Dropbox Tech, November 3, 2022. Companion: 5 Lessons from 5 Years of Building Databases at Scale, Sammy Steele, Systems Distributed '24.
GitHub stores its metadata in MySQL. Issues, pull requests, comments, organizations, notifications. One master takes the writes. Replicas serve the reads. A dead master used to mean a human waking up and running a failover by hand. That was too slow for the availability they wanted.
One master, replicas, one watcher
Detection plus a flap guard, not just a tool. Automation without a guardrail is a second outage waiting to happen.
They run Orchestrator. It watches the replication topology and runs the failover itself. Their target was detection plus recovery in 30 seconds or less, with minimal loss of hosts. Detection is holistic. It asks the replication cluster what happened instead of trusting one monitor. They tested its decisions for months before they turned on automatic recovery.
The guardrails matter as much as the tool. Flapping is not allowed. Orchestrator runs at most one automatic failover per cluster in a set time period. After a failover, the cluster is blocked from another one for 30 minutes, or until a human acknowledges it. It also handles a dead intermediate master, so a middle layer failing does not strand the replicas below it. Later work added Pseudo-GTID handling for replicas that fall behind or go unreachable.
My take: the failover tool is only half the story. The other half is the policy around it. One failover, then a block, then a human. Automation without a flap guard is a second outage waiting to happen. I check for that guardrail in any failover tool before I trust it.
Source: Orchestrator at GitHub, The GitHub Blog.
The fleet was 1,200-plus MySQL hosts, a mix of Azure VMs and bare metal. They held 300-plus TB of data and served 5.5 million queries per second across 50-plus clusters. MySQL 5.7 was near end of life, so they moved to 8.0. Planning, testing, and the upgrade took over a year.
The upgrade, in four steps
Jul 2022: 8.0 added to CI. 5.7 and 8.0 side by side, so regressions showed up before production.
Dual chains. 8.0 replicas online, one data center at a time, behind 5.7 primaries kept as rollback.
Promote. An 8.0 replica becomes primary through Orchestrator. No primary was ever upgraded in place.
24-hour validation. Then the 5.7 rollback path closes. The collation change, roles, and one replica bug bit along the way.
Prep started in July 2022. They added 8.0 to CI and ran 5.7 and 8.0 side by side, so regressions showed up before production. They never upgraded a primary in place. They brought 8.0 replicas online, one data center at a time, and promoted an 8.0 replica to primary through Orchestrator.
The rollback path was the design, not an afterthought. They kept 5.7 standbys behind the 8.0 primaries, so they could roll back without a disruption. Each cluster got a 24-hour validation cycle before the rollback path closed. The collation change bit them. Version 8.0 defaults to utf8mb4_0900_ai_ci, and 5.7 does not understand it, so backward replication broke. Roles broke replication too. A replica_preserve_commit_order bug, fixed in 8.0.28, was another one. A WHERE IN list with tens of thousands of values crashed MySQL.
My take: the upgrade was a replication project, not a version bump. If you cannot roll back, you cannot upgrade a fleet this size. I would copy the shape. Dual chains, one cluster at a time, a fixed validation window, and the rollback kept alive until the window closes.
Source: Upgrading GitHub.com to MySQL 8.0, The GitHub Blog, December 7, 2023.
Slack started moving its MySQL datastores to Vitess in 2017. By December 2020, 99 percent of all MySQL traffic ran on Vitess. Peak load was 2.3 million queries per second: 2 million reads, 300 thousand writes. Median latency was 2 ms. p99 was 11 ms.
Workspace sharding vs channel ID resharding
One big customer lands on one shard. It runs hot while the rest sit idle.
Traffic spreads. In March 2020 query rates rose 50 percent in one week. They split a busy keyspace live and kept serving.
Sharding is a routing problem before it is a capacity problem.
The old shape was workspace sharding on active-active MySQL. It worked until a big customer landed on one shard and made it hot while the rest sat idle. Enterprise Grid and Slack Connect broke the model further, because one user's data now spanned workspaces.
The move was a re-shard by channel ID under Vitess. The migration ran on double-write backfill plus double-read diffing. Write to both, read from both, compare, and only then cut over. In March 2020, query rates rose 50 percent in one week. They split a busy keyspace horizontally and kept serving. Without resharding, they wrote, their largest customers would have had downtime.
The extra win came later. The same Vitess clusters now run their International Data Residency product in six regions. My take: sharding is a routing problem before it is a capacity problem. The channel key fixed the hot spot that the workspace key created. Pick the key from the access pattern, and build the migration so you can compare both copies before you trust one.
Source: Scaling Datastores at Slack with Vitess, Slack Engineering, December 1, 2020.
The Shop app backend is Ruby on Rails on a single MySQL, their managed system called KateSQL. Growth was steep. Disk grew to many terabytes. Schema migrations took weeks. When the database got busy, they throttled background jobs.
The five verifiers, checked before any data moved
A commit that lands on one shard and fails on another does not roll back the first. Build the check first, move the data second.
The first fix was federation. GhostFerry moved whole tables onto new databases, and new domains got their own databases. That bought time and limited blast radius. It hit its own wall. Migrations on big tables still took weeks, joins across databases got hard, and the app had to know where every table lived.
The second fix was Vitess, sharded by user_id. User-owned data went into a sharded users keyspace. Everything else went into an unsharded global keyspace. Before moving, they built query verifiers into the app layer. The verifiers caught queries missing the sharding key, transactions crossing databases, transactions crossing shards, writes crossing shards, and queries joining across keyspaces. They removed cross-shard transactions entirely. A commit that lands on one shard and fails on another does not roll back the first. Moving a row across shards became two steps: insert in the new shard, then delete from the old one.
My take: the verifiers were the project. Shopify wrote that if they could credit one thing for the move working, it was the correctness of the verifiers. Sharding without a check that every query carries its key is a data corruption incident on a schedule. Build the check first, move the data second.
Source: Horizontally scaling the Rails backend of Shop app with Vitess, Shopify Engineering, January 17, 2024.
One story so far, and it is the classic MongoDB failure: the working set outgrows RAM, the cache stalls, and the connection storm finishes the job. The fix is timeouts, throttling, and reads off the primary, not a bigger instance.
On April 11, 2018, Fortnite shipped version 3.5, and logins stopped working for all players. The MongoDB database behind the account service had degraded until the account service was unavailable, from 20:44 UTC on April 11 to 14:30 UTC on April 12. A change in call patterns in 3.5 slowly pushed cache reads up over hours. The dataset no longer fit in RAM, so pages were evicted and read back from disk faster and faster, until evictions ran 4.5 times above baseline. Dirty cache, data written to RAM and not yet to disk, built up to about 29 GB, and the primary could no longer keep up with both request load and replication load. The cache pressure was enough to overload the primary and trigger leader elections.
The cascade: slow queries became a connection storm
Short timeouts turned slow queries into new connections. New connections turned a slow database into an unreachable one.
They tried to bring services back seven times over 17 hours, and each attempt ran into the same cascade. The fixes that stuck: the socket timeout went from 200 ms to 30 seconds, so the app stopped closing connections at the first slow query and churning new ones. Queries got a 500 ms maxTimeMS instead, so the database kills a slow operation and keeps the connection open. The app moved behind mongos, the shard router, which has more connection controls than the driver, plus a throttle on opening new connections to the database, about 50 at a time. A code path that pinned specific reads to the primary was changed to read from secondaries.
The Friend Suggestions bug, re-querying the list on every friends list refresh, had pushed one API call to 300 times its normal load, and the hotfix tunables meant to control it were broken. Version 3.5.1 fixed both. Ephemeral data, short-lived single-use OAuth authorization codes, moved out of MongoDB into Redis. Better caching of account data cut database lookups at least in half. Recovery meant removing traffic and adding clients back a little at a time from a cold start. Two replica set members were never explained: their CPU hit 100 percent, almost all of it in kernel time, logging stopped, and the process looked hung. Those two were isolated and replaced.
The storm was the killer, not the data size. A bigger instance would have held more cache, but the connection churn would still have knocked nodes offline once the database slowed. Throttling connections at mongos protected the replica set's own communication, which is what kept members from dropping out.
The part I would copy is the connection throttle. When the database slows down, the client must not be allowed to make it worse.
Read this when your timeouts are short and your connections spike under load.
Source: Postmortem of Service Outage 4/11/2018 - 4/12/2018, Fortnite (Epic Games), April 26, 2018.
Before you leave, check this: pick the story closest to your own system, and name the one decision in it you have not made yet. The shard key, the failover guardrail, the rollback path, the index shape. That decision is the incident you have not had yet. The hosting side of the same question, on-prem, EC2, Kubernetes, or managed, is on the Cloud page.