Status: Resolved
Incident Date: 14 July 2026
Duration: ~11:10 UTC – ~20:41 UTC (approx. 9.5 hours from first production impact to recovery)
Reference: For the exact timeline, please reference our status page.
Speech Engine conversations failed to establish and closed with an error indicating the agent could not be found.
Impact ramped up as the faulty release progressed through its staged rollout. At full rollout, all Speech Engine conversation traffic was failing to connect. Standard agent conversations remained unaffected throughout the duration of the incident.
A recent change to how agent permissions are verified introduced a new permission check into the conversation setup path. That check assumed every conversation belonged to a standard agent.
Speech Engine conversations are run by a different type of resource whose prompt is not controlled on our platform, causing the resource load to fail and the check to reject the connection.
Detection was delayed because the unintended behavior surfaced as a client-side error. As a result, the issue was only surfaced by our support team roughly six hours after production impact began.
Releases are validated in staging, then rolled out gradually through a staged rollout, meaning each release's effect ramps up until full deployment.
10:01 - The release containing the faulty change is published.
11:10 - The release begins rolling out to production. Speech Engine conversations begin failing and ramp up toward broad production traffic as the rollout progresses. No automated alert fires.
17:06 - The incident is officially declared following escalations from our support team and the root cause is identified immediately.
18:08 - Following our standard release process, a targeted hotfix is developed and deployed.
20:41 - The fix release begins rolling out to production. Speech Engine conversations fully recover as the rollout progresses.
The following engineering follow-ups are already underway to prevent recurrence and improve time-to-detection:
Improve Alerting Systems: Re-enable and redesign "agent not found" alerting so abnormal spikes in client side errors and connection failures page the team.
Expand Test Coverage: Add more Speech Engine test coverage in staging so this class of issue is caught before release.
Review Permission Checks: Revisit how permission checks account for different product surfaces, ensuring a check written for one agent/product type cannot inadvertently break another.
