Enjoyable with incident knowledge and statistical course of management – Browsing Complexity

Last yr, I wrote a submit referred to as TTR: the out-of-control metric. In that submit, I argued that the incident response course of(specifically, the time-to-resolution metric for incidents) won’t ever be beneath statistical management. I confirmed two notional graphs. The first one was indicative of a course of that was beneath statistical management:

The second graph confirmed a course of that was not beneath statistical management:

And right here’s what I stated about these graphs:

Now, I’m prepared to wager that for those who have been to attract a management chart for the time-to-resolve (TTR) metric in your incidents, it might look much more just like the second management chart than the primary one, that you simply’d have a variety of incidents whose TTRs are nicely exterior of the higher management restrict.

I believed it might be enjoyable to try some precise publicly out there incident knowledge to see what a management chart with incident knowledge truly appeared like. Cloudflare’s been on my thoughts today due to their latest outage so I believed “hey, why don’t I take a look at Cloudflare’s data?” They use Atlassian Statuspage to host their standing, which features a history of their incidents. The good factor about Statuspage is that for those who move the Accept: utility/json header to the /historical past URL, you’ll get again JSON as a substitute of HTML, which is handy for evaluation.

So, let’s check out a management chart of Cloudflare’s incident TTR knowledge to see if it’s beneath statistical management. I’m going into this realizing that my outcomes are more likely to be extraordinarily unreliable: as a result of I’ve no first-hand knowldge of this knowledge, I don’t know what the connection is between the time an incident was marked as resolved in Cloudflare’s standing web page and the time that clients have been not impacted. And, generally, this timing will differ by buyer, but one more reason why utilizing a single quantity is harmful. Finally, I’ve no expertise with utilizing statistical course of management methods, so I’ll simply be plugging the info right into a library that generates management charts and seeing what comes out. But knowledge is knowledge, and that is only a weblog submit, so let’s have some enjoyable!

Filtering the info

Before the evaluation, I did some filtering of their incident knowledge.

Cloudflare categorizes every incident as certainly one of vital, main, minor, none, upkeep. I solely thought of incidents that have been categorized as both vital, main, or minor; I filtered out those labeled none and upkeep.

Some incidents had extraordinarily giant TTRs. The 4 longest ones have been 223 days, 58 days, 57 days, and 22 days, respectively. They have been additionally all billing-related points. Based on this, I made a decision to filter out any billing-related incidents.

There have been a variety of incidents the place I couldn’t mechanically decide the TTR from the JSON: These are instances the place Cloudflare has a single replace on the standing web page, for instance Cloudflare D1 – API Availability Issues. The period is talked about within the resolve message, however I didn’t undergo the extra work of making an attempt to parse out the period from the pure language messages (I didn’t use an AI doing any of this, though that may be an excellent use case!). Note that these aren’t at all times quick incidents: Issues with Dynamic Steering Load Balancers says The surprising behaviour was famous between January thirteenth 23:00 UTC and January 14th 15:45 UTC, however I can’t inform in the event that they imply “the incident lasted for 16 hours and 45 minutes” or they’re merely referring to after they detected the issue. At any price, I merely ignored these knowledge factors.

Finally, I checked out simply the 2025 incident knowledge. That left me with 591 knowledge factors, which is a surprisingly wealthy knowledge set!

The management chart

I used the pyshewhart Python bundle to generate the management charts. Here’s what they appear like for the Cloudflare incidents in 2025:

As you possibly can see, this can be a course of that’s not beneath statistical management: there are a number of factors exterior of the higher management restrict (UCL). I notably take pleasure in how the pyshewhart bundle superimposes the “Not In Control” textual content over the graphs.

If you’re curious, the longest incident of 2025 was AWS S3 SDK compatibility inconsistencies with R2, a minor incident which lasted about 18 days. The longest main incident of 2025 was Network Connectivity Issues in Brazil, which lasted about 6 days. The longest vital incident was the one which occurred again on Nov 18, Cloudflare Global Network experiencing issues, clocking in at about 7 hours and 40 minutes.

Most of their incidents are considerably shorter than these lengthy ones. And that’s precisely the purpose: a lot of the incidents are temporary, however each on occasion there’s an incident that’s for much longer.

Incident response won’t ever be beneath statistical management

As we will see from the management chart, the Cloudflare TTR knowledge will not be beneath statistical management, we see clear cases of what the statisticians Donald Wheeler and David Chambers name distinctive variation of their e book Understanding Statistical Process Control.

For a course of that’s not beneath statistical management, a pattern imply like MTTR isn’t informative: it has no predictive energy, as a result of the method itself is essentially unpredictable. Most incidents is perhaps quick, however then you definitely hit a very robust one, that simply takes you for much longer to mitigate.

Advocates of statistical course of management would inform you that the very first thing you must as a way to enhance the system is to get the method beneath statistical management. The grandfather of statistical course of management, the American statistician Walter Shewhart, argued that you simply needed to establish what he referred to as Assignable Causes of remarkable variation and tackle these first as a way to eradicate that distinctive variation, bringing the method beneath statistical management. Once you probably did that, then you could possibly then tackle the Chance Causes as a way to cut back the routine variation of the system.

I believe we must always take the lesson from statistical course of management {that a} course of which isn’t beneath statistical management is essentially unpredictable, and that we must always reject using metrics like MTTR exactly as a result of you possibly can’t characterize a system out of statistical management with a pattern imply.

However, I don’t suppose Shewhart’s proposed method to bringing a system beneath statistical management would work for incidents. As I wrote in TTR: the out-of-control metric, an incident is an occasion that happens, by definition, when our methods have themselves gone uncontrolled. While incident response might continuously really feel prefer it’s routine (detect a deploy was unhealthy and roll it again!), we’re coping with complicated methods, and complicated methods will sometimes fail in complicated and complicated methods. There are much more ways in which methods break, and the distinction between an incident that lasts, say, 20 minutes and one which lasts 4 hours can come down as to whether somebody with a related bit of information occurs to be round and might deliver that data to bear.

This truly will get worse for extra mature engineering organizations: the extra dependable a system is, the extra complicated its failure modes are going to be when it truly does fail. If you attain a state the place your entire failure modes are novel, then every incident will current a set of distinctive challenges. This signifies that the response will contain improvisation, and the time will rely upon how nicely positioned the responders are to take care of this unexpected state of affairs.

That being stated, we must always at all times be striving to enhance our incident response efficiency! But regardless of how a lot better we do, we have to acknowledge that we’ll by no means be capable to deliver TTR beneath statistical management. And so a metric like MTTR will eternally be ineffective.