Methodology
How we score software reliability
Our rankings answer one question for business owners: which vendor is least likely to leave you unable to trade? Every number on this site comes from two sources of data, combined by a published formula. Nobody can pay to change it.
1. Data we collect
Vendor status pages
Every five minutes we read each vendor's public status page (Atlassian Statuspage, incident.io, Instatus and others) and record its current state. Every 30 minutes we pull its incident history: when each incident started, when it was resolved, its severity and which components were affected. Every response is logged, so we keep the full history even if a vendor later edits or deletes an incident.
Our own uptime checks
Status pages are written by vendors, so we don't rely on them alone. Every five minutes our probes request each vendor's public login page and customer-facing pages (online ordering, booking or payment pages) from one or more regions. A check counts as failed on a network error, a timeout or a 5xx server error. Responses like 403 or 429 (bot protection, rate limiting) are logged as inconclusive and never counted as downtime. An outage opens after two consecutive failed checks from the same region and closes at the next successful check. If most vendors fail at once from one probe, we assume the problem is ours and discard those results.
Shared status pages
Some companies publish one status page for several products (for example Square's point of sale, payroll and online store). Where the page says which components an incident affected, we only count it against the products involved. Where it doesn't, we can't rule a product out, so the incident counts against every product on that page.
2. The reliability score
Each vendor gets a score from 0 to 100, calculated over a rolling 90-day window from four components:
| Component | Weight | What it measures |
|---|---|---|
| Uptime | 40% | Effective uptime after weighting incidents by severity |
| Incident frequency | 20% | How often things go wrong, per 30 days |
| Time to resolve | 20% | Average time from start to resolution |
| Transparency | 20% | How often the vendor acknowledges outages we detect |
Uptime (40%)
Not every incident takes a product fully offline, so each minute of an incident counts as a fraction of a minute of downtime:
| Severity | Counts as |
|---|---|
| Major outage (critical) | 100% of each minute |
| Partial outage (major) | 50% |
| Degraded performance (minor) | 15% |
| Outage detected by our probes | 100%, scaled by the share of probe regions that saw it |
| Scheduled maintenance, informational notices | 0% |
Where incidents overlap, including when the vendor's status page and our probes report the same outage, only the most severe one counts, so nothing is double counted. Incidents are capped at 72 hours so one incident a vendor forgot to close can't dominate the score. The uptime component scores 100 at 100% uptime, 90 at 99.9% and 50 at 99.5%, then falls more gently: 37.5 at 99%, 25 at 98.5% and 0 at 97.5% or below.
Incident frequency (20%)
We count incidents that started in the window, weighted by severity (major outages and partial outages count 1, degraded performance 0.5, informational notices 0.25, maintenance 0), normalise to a 30-day rate and score it as 100 × e^(−rate ÷ 4). One incident a month scores about 78. Four a month scores about 37.
Time to resolve (20%)
The average time from the start of each resolved incident to its resolution, scored as 100 × e^(−hours ÷ 12). A 30-minute average scores about 96, a 6-hour average about 61. Vendors with no resolved incidents score 100.
Transparency (20%)
When our probes detect an outage of at least 10 minutes, we check whether the vendor posted an incident on its status page that overlaps it (within 15 minutes). The transparency score is the smoothed share of detected outages that were acknowledged, so a single miss doesn't zero a vendor. Vendors without a public status page score 0 on transparency, because customers can't see when they are having problems.
3. Rankings
Vendors are ranked within each category by score, with uptime as the tie-breaker. A vendor only joins a ranking once we have at least 14 days of data for it. A quiet status page is not treated as proof of reliability: if a vendor has posted nothing for a year, or nothing for 90 days in a category where the typical vendor reports three or more incidents, it stays unranked until our own monitoring has watched it for 30 days. Where a vendor resolves incidents without saying when, those incidents count towards frequency but not towards downtime, and its time-to-resolve score is neutral (50) rather than perfect. For vendors with a status page, we import its published incident history, so most vendors start with months of data. Where a status page only exposes its most recent incidents, we only score the period that history fully covers.
4. Limitations
- Vendors that report incidents in great detail can look worse than vendors that report nothing. Our own monitoring and the transparency component exist to correct for this.
- Our probes see public pages, not every feature. A vendor's login page can load while card payments are failing. Status page incidents cover what our probes can't see.
- Severity labels are chosen by vendors and aren't perfectly consistent between companies.
5. Independence
We earn money from affiliate links and clearly labelled sponsored placements. Neither affects scores or rankings: the formula above is applied to every vendor the same way, and sponsored placements are shown separately from the rankings. See our disclosure policy.