EU Residency ElevenAgents Outage (September 28, 2026)
Duration:
September 28, 2026, 1:40 PM UTC - September 28, 2026, 2:18 PM UTC
Description of the Issue:
Most components of ElevenAgents were unresponsive, including placing calls and managing agents.
Root Cause:
The managed MongoDB cluster dedicated to EU ElevenAgents became unresponsive due to OOM errors caused by a spike in search-based database traffic. The managed cluster was unable to recover automatically and required manual intervention from the MongoDB support team.
Timeline & Actions Taken:
September 28, 2026 1:48 PM UTC - Automatic cloud alerts page our engineering team regarding an elevated rate of requests returning 500 status codes. An investigation begins immediately.
September 28, 2026 1:53 PM UTC - The incident response team declares a P0 critical incident and confirms an ongoing outage.
September 28, 2026 1:57 PM UTC - Our status page is updated to reflect that we are investigating an incident.
September 28, 2026 1:58 PM UTC - The response team confirms the issue originates from our managed MongoDB cluster.
September 28, 2026 2:05 PM UTC - Following initial investigation, the team determines that no direct action on the ElevenLabs side can recover the cluster. Our nodes are down and fail to self-recover. ElevenLabs pages MongoDB support with maximum incident severity.
September 28, 2026 2:07 PM UTC - The MongoDB support team starts investigating internal logs.
September 28, 2026 2:10 PM UTC - The MongoDB engineering team starts manually rebooting the cluster.
September 28, 2026 2:18 PM UTC - The cluster is fully back up and healthy. Customers stop seeing errors.
Preventative Actions & Learnings:
ElevenLabs and MongoDB are working together to implement the following follow-up actions:
Dedicated search nodes to isolate the impact of sudden spikes in high-fluctuation search traffic.
Preventative memory usage alerting for MongoDB hosts.
Memory-based auto-scaling for MongoDB hosts.
