Runtime outage impacting workspace authentication
We’re seeing elevated errors for login requests across US workspaces. Follow this thread for mitigation updates.View status page
Active incident
Major outage

Authentication downtime in US-West

Teams cannot issue new session tokens while we revert the worker deploy. Existing sessions stay valid.

Updated 6 minutes ago

Impact

Login + token refresh

Reference

INC-9824

Affected regions

US-WestUS-Central
Status history

Incident timeline

Recovery milestone tracker

3 updates
Issue identified
02:14 UTC
Critical

Edge workers in the us-west cluster began rejecting auth tokens after a configuration rollout.

  • Rollback in progress with 85% of nodes drained
  • Customer auth latency spiked to 6.2s
Mitigation applied
02:36 UTC
Monitoring

Zero-downtime patch pinned the previous worker build and redirected traffic through healthy regions.

  • Latency trending back under 400ms
  • API success rate steady at 99.2%
Resolved
03:05 UTC
Resolved

All regions report healthy signals. We will continue monitoring till the postmortem is published.

  • Root cause isolated to stale config in infra pipeline
  • Retro scheduled with runtime guild

Live notes from the response channel

Engineers reverted the faulty worker build and locked deployments until the fix is verified. No customer data was lost; requests received 5xx responses during the 12-minute fault window.

We are shipping a follow-up deploy with additional safeguards for configuration diffing and have added extra telemetry for token validation.

Incident command center

Shortcuts to runbooks, documentation, and escalation steps.

Retry attempts: 0

Background refreshes: 0