Outage report · Major outage

Convex outage on Mar 4, 2026

high error rates on customer backends

SeverityMajor outage

StartedMar 4, 2026, 17:28UTC

ResolvedMar 4, 2026, 18:28UTC

Duration1h

What happened

Convex reported a major outage starting Mar 4, 2026, 17:28 UTC, affecting Free & Starter. It was marked resolved at Mar 4, 2026, 18:28 UTC.

The incident lasted 1h from first report to resolution.
Convex had reported 3 other partial or major outages in the 90 days before it.
3 other vendors we track reported a significant incident within 12 hours of it.

Latest update from Convex

Convex had downtime for some customers caused by an anomalous load spike that pushed an internal service into a load-shedding mode, which then triggered an unexpected panic in a caching library we were using. Specifically the load triggered two queue management algorithms: CoDel which proactively drops requests to keep queues small, and adaptive-LIFO which dequeues in reverse order to avoid wasting time on old requests. These are both rather subtle algorithms that large services use to avoid congestion collapse under high load or attack. The panic in the caching library was just a bug that depended on both these algorithms simultaneously. We've made some changes as a result of this incident but the key lesson is that services should try to avoid switching logical behavior during high load. When systems are stressed switching to infrequently-used codepaths can often make matters worse. We're now going to be proactively triggering CoDel and adaptive-LIFO at steady state to ensure that

Read Convex's official incident report ↗