Universe outage on Aug 9, 2022

What happened

Universe reported a partial outage starting Aug 9, 2022, 06:44 UTC, affecting Ticket Processing. It was marked resolved at Aug 9, 2022, 21:13 UTC.

• The incident lasted 14h 30m from first report to resolution.
• Universe had reported 1 other partial or major outage in the 90 days before it.

Latest update from Universe

Just before 1 am ET one of the databases we use for tracking payments experienced an out of memory error. This delayed the final step in the automatic capture of credit card funds for the duration of the outage. Once service was restored, payment collection automatically resumed, retroactively finalizing transactions created during the outage period. While we have extensive monitoring in place to notify us in advance of disruptive conditions, we’ve determined that the alarm for this particular metric was the incorrect type. For that reason, the impending outage was very easy to miss. The database disruption caused upstream health checks to fail, which were immediately surfaced to the entire dev team in Slack. With the problem identified, memory was increased for the payment tracking database and service was quickly restored. We take reliability & stability very seriously at Universe. While our already robust infrastructure allowed us to recover from this outage without dropping any

Read Universe's official incident report ↗

What our independent monitoring saw

Our independent monitoring of Universe started after this incident, so this report is based on Universe's status page.

Universe's overall reliability

Full Universe reliability report and live status → · Most reliable Universe alternatives →