Between 10:29 AM UTC and 2:59 PM UTC on July 2, calls made through Twilio phone numbers experienced failures due to a recent update, leading to customers reaching call capacity and encountering 429 errors. SIP phone numbers using Twilio SIP Trunks were unaffected.
The problematic update was rolled back at 12:54 PM UTC, stopping new failures. Full call capacity was restored by 2:59 PM UTC. We are implementing improved testing, telemetry, and monitoring to prevent recurrence and enhance response times.
Resolved
Between 10:29 AM UTC and 2:59 PM UTC on July 2, calls made through Twilio phone numbers experienced failures due to a recent update, leading to customers reaching call capacity and encountering 429 errors. SIP phone numbers using Twilio SIP Trunks were unaffected.
The problematic update was rolled back at 12:54 PM UTC, stopping new failures. Full call capacity was restored by 2:59 PM UTC. We are implementing improved testing, telemetry, and monitoring to prevent recurrence and enhance response times.
Monitoring
We've rolled back our latest release, and the system seems to be holding steady, which indicates a faulty release. We're still investigating the main cause of the issue, but we have a good idea of what it is.
Identified
We've identified the cause of the outage and are working on finding a resolution. We'll likely roll back as we investigate.
Investigating
We've noticed an issue with Twilio phone number calls failing, and it has been escalated internally with priority for investigation by the engineering team.
