Isolated environments (EU + India) - TTS with Voice Library voices failing

Duration:
October 31, 2025, 00:08 UTC - October 31, 2025, 06:31 UTC

Description of the Issue:

Customers were unable to use Voice Library voices for TTS in isolated environments (India, EU), due to a release. Over the course of the incident, requests gradually started failing as voices became inaccessible for use in text to speech. 

Root Cause:

The incident was caused by a change made to how we cache user details of Voice Library voices in data residency environments. This incorrect change made isolated environments unable to fetch any Voice Library voices from our US cluster. As long as the voices were still cached, TTS was able to run smoothly but slowly all these requests started erroring as the cache for these voices were getting invalidated.

Timeline & Actions Taken:

00:03 UTC: The faulty change starts rolling out

00:08 UTC: TTS errors for Voice Library voices start appearing

03:44 UTC: Engineering paged by cloud alerts. At this point the issue seemed to be limited to one voice and one account so the issue was postponed until UTC morning.

05:16 UTC: Engineering paged again by cloud alerts. Incident is declared. 

05:45 UTC: The backend is rolled back to a previous version. Existing poisoned caches are still causing some errors. The engineering team is trying to understand why.

05:58 UTC: Root cause is identified.

06:31 UTC: Caches are manually cleared by the engineering team. The issue is remediated. 

Preventative Actions & Learnings:

Learnings: The issue could have been caught by a test environment which simulates an isolated environment fully, as we learned that our current test environment is not identical to the EU and India environments. 

Preventative Action: Build a test environment that mirrors the exact configuration and data access constraints of the EU and India clusters so that any issues such as this would be caught in the automated testing phase prior to release.