Description of the Issue:
A MongoDB database that stores data for our Agents Platform became unresponsive in our US cluster following a database maintenance operation. This slowed down or blocked many requests, and during the incident roughly 45% of US calls failed and 7% of API requests returned errors.
Root Cause
The unresponsiveness was triggered by a background database maintenance operation (MongoDB expected to provide detailed response as routine operation should not lead to interruption). The database’s built-in protections kicked in as designed, but they stalled writes instead of recovering on their own, and the failure repeated each time we switched to a healthy server.
Fixes only held briefly because the underlying pressure remained. We worked with our database vendor, MongoDB, to stabilise the cluster, and added capacity to bring it back to full health.
Preventative Actions & Learnings:
Retrospective with MongoDB team on resilient changes required on configuration
Coordination with MongoDB on how background database operations are scheduled and approved.
Additional alerting on database health indicators (queue depth, cache pressure, connection counts) that showed strain before users see errors
We’re moving data that Agents Platform calls depend on onto a dedicated, smaller cluster, so it’s less exposed to load elsewhere.
Multi-region resilience. We’re replicating this cluster across multiple US regions, reducing the chance that a single region or server failure causes downtime.
Timeline:
08:50: Impact began.
08:54: An automated alert fired for a high ElevenAgents API error rate, and the on-call team began investigating.
09:10: Database query timeouts were identified, pointing to a database-side issue.
09:21: A support case was opened with MongoDB.
09:25: An overloaded database server that was stalling writes was identified.
09:32: Service briefly recovered, then failed again at 09:46.
09:59 A working call began with MongoDB engineers.
10:15 Traffic was manually switched to a healthy database server, and service briefly recovered at 10:19.
10:43 Failures returned, and the original server was still unable to recover.
11:06: Background shard rebalancing was paused to reduce load.
12:08: Service briefly recovered after MongoDB manually brought the original server back online.
12:26: Failures returned.
12:27: A capacity upgrade was started.
12:45: The upgrade was delayed by cloud provider capacity limits in the region, and MongoDB began manually restarting and repairing the stuck servers.
13:14: Service fully recovered.
14:18: The remaining servers were upgraded one at a time, with traffic first moved to a healthy server.
14:34: All servers were back online.
