When Good AI Goes Bad in Your IT Services Provider

I once had a friend who spent half his time getting himself into trouble and the other half trying to get himself out. This approach is how some companies enter marketing an artificial intelligence offering. Let me explain.

Developing Artificial Intelligence

As most companies are beginning to entertain the idea of adding artificial intelligence (AI) into their services, I imagine the conversations surrounding data science take place much like studio executives evaluating a script. Imagine sitting in a room where an incredible script for one of the best western movies of all time is being discussed. After the pitch, there’s no question this is going to be a hit; it offers a new take on classic themes that appeal to a dedicated audience. As everyone nods in excitement, eager to start production, there’s an unexpected 180.

 One of the executives says, “It’s a great start. But you know what would make this movie even better?” With a smile and a nod, he says, “Aliens.” Before anyone else can get a word in, the executive edits his own idea, in real time, and with an even larger smile adds, “And monsters.” And this is how a perfectly good film gets instantly turned into something else. I suspect this happens a lot when companies discuss AI, as well. We all think we’re agreeing to one narrative only to have several others elbow their way in, making what would be a blockbuster into an expensive bust.

Use Data Science to Evaluate the Need for Building Artificial Intelligence

Good development around an AI project is just as important as good development in filmmaking or any project that requires a large number of resources to build. AI only works if you know what questions you intend to have it solve. A company needs to agree on what story they are hoping to build before they start production.  

Sure, some people might think aliens are a cool idea, but do they really need to be added to a classic western’s script to make it good? Or are they an expensive distraction? Was their inclusion driven simply because someone liked aliens? Shouldn’t the goal of the whole project be considered and agreed to before such a large addition is signed off on? The same questions apply to a business that has recently added AI as a feature. Why does the final product look the way it does?[1]

This is why while a company is busy selling you AI, it is incredibly helpful to understand their process in developing it. Did they use data science to evaluate the need for artificial intelligence? If any company is sitting around talking about how cool it would be to have AI do this or that without data to inform “this” or “that” it probably has been built for their own entertainment and not necessarily to solve your problems.

This is something worth considering in any industry, but especially for one that take considerable resources to achieve, like an artificial intelligence offering. If you don’t center your product on your audience, then what kind of outcome are you really delivering?

AI Might Be Solving the Wrong Problem, And That’s A Problem

A well-known and leading service delivery company recently released a marketing campaign to demonstrate their new AI development. They are working to solve the problem of missed or delayed diagnosis. They are attempting to solve a delay problem through machine learning which automatically detects failures in an auto support log file. This is an incredibly sophisticated solution for a problem that plagues most service delivery companies. An enormous amount of time and money was reported to have been spent solving this problem. But I was left asking the question, “Is this an adequate solution to resolve delays?”

I know this company well and have spent a lot of time reviewing their service delivery reports.  And to be fair, it’s easy to sit back in an armchair empire and pick apart a company’s solutions. But when building out an artificial intelligence offering, we need to start by understanding how to correctly use data science to evaluate a problem. Part of that means we need to come to the table humble enough to have these hard conversations. We need to collectively develop the idea to succeed. Sometimes that’s going to mean leaving the alien monsters on the drawing board.

The well-known company I want to highlight has a problem with delaying service delivery. But the important insight they’ve missed is that the majority of those delays are not caused by a delayed diagnosis. Their AI is an impressive production on its own but unfortunately it’s solving for the wrong problems.   

Understand the Problems AI Should Be Used to Solve

Below is a chart to visually highlight how CloudCover goes about analyzing problems before we build a solution. It gives you some insight into our development process.

This chart graphs the amount of time the average of cases sat in a particular status. Each of these cases are from the last year of services. Recently we began using a custom-built AI to help manage this partner as we worked to better understand their problems. The result can be seen through a month by month decrease in overall delays.  

The chart identifies different cycles in a case. At first glance, you might be tempted to think that delays are mostly caused by the diagnosis taking too long. But that’s not what’s happening.

After evaluation, we found that “diagnosis” was not the primary cause of delays. The “part order,” “scheduling a technician,” and “customer caused delays” took up the majority of the case lifecycle.  Below is the math used to evaluate the sums of status. Using data science to understand where the problem really resided meant we could build a solution that actually addressed solving for it.

At CloudCover, we are not in the business of simply solving problems. We use Data Science to prove that our hypothesis is correct about the problem, first. Once we thoroughly understand the problem, we move to solve it. Details are always important but getting too involved in singular details (i.e. focusing only on the timetable for diagnosing and none of the other parts) sometimes leads to the old “can’t see the forest for the trees” problem. We aim to always make sure we’re seeing the whole forest, rather than just a single tree.

Break Down Complex Problems into Smaller Data Sets to Review

When you are dealing with global IT maintenance you are almost always dealing with complex problems. In order to correctly address them, each problem requires sophisticated analysis from multiple disciplines.

Let’s continue to use the example I outlined above. In this type of example, there are three areas of different problems presented within the average case to review:

1. Diagnosing
2. Part and Tech Delivery
3. Customer Caused Delays

Each of those three problems require intense analysis because the three problems are not caused by the same thing. While we began analyzing the diagnostic delays, some interesting results were noted. Among the observations two really stood out as ones to explore and understand further:

1. Within the mixture of diagnostic delays, easy to diagnose problems were just as likely to cause delays as complex problems.
2. Although an L1 team should be capable of diagnosing a simple failure or assessing if a customer diagnosis is accurate, a delay was still created.

To Find a Solution First Allow Room for Complex Problems

 It’s important to acknowledge that there are myriad of complexities surrounding delays. To find an adequate solution you must be willing to allow for a complex problem. This means not stopping at what appears to be the first solution. It means looking for ways to tear that solution apart to see if it holds under analysis. It means being okay with possibly being wrong a couple times in order to be confidently right. This development work is crucial when creating an AI solution. In ensures the AI is solving the correct problem.

Not All Service Delays are Equal

“Diagnostic delays” can be connected to a company’s Service Level Agreements (SLA). SLAs defines the rules for how a company should proceed through a service call. When a company requires a part or technician within four hours, often the time for that four hours does not begin until after the diagnosis is made.

Companies have been known to purposely delay a diagnosis because they do not have the part or engineer available for dispatch within four hours. The delay is designed to preserve the SLA adherence. This is a not a justifiable reason to delay a case and producing a machine learning diagnosis is not a valid solution for this type of problem.

Auto Diagnostics versus Artificial Intelligence

Our solution is a bit different than just auto diagnostics, although we use this as a tool to quickly manage cases. Our goal is to build a high-level approach to resolve all delays. We begin before we even build a quote for the equipment. Ensuring part and engineering availability is key to managing a four hour SLA.  

We use machine learning to predict the probability of each potential failure. We then build inventories, and our engineering requirements based on this machine learning algorithm. Our ultimate goal is to identify and approach potential delays before they ever happen. 

Data isn’t static so it’s important that we don’t just create a solution and let it run. We check our own work, constantly. We allow for those complex problems I mentioned earlier and the work it takes to identify and solve for them.

We approach this methodology in a couple very specific ways. First, the potential for delays is measured through our CloudCover portal, with each status independently and automatically assessed as a case moves through its lifecycle. We then automatically engage the AI to evaluate the context of a potential delay and produce varied solutions for resolving it. Those solutions are then managed and measured to continue building better support models. This practice of continuously measuring and evaluating solutions helps us build better resolutions.

Want to see how this works in a real world example? Below is the analysis from a previous AI build against another provider who began being managed by full AI.

The orange line demarks the AI working almost exclusively. It performs at a flatter level which indicates less delay. The AI is reducing all delays — including the customer caused delays.

AI Develops with Data Science

The bottom line is this:  without a consistent data science practice in place around their AI’s development, a company simply cannot accurately identify problems. Without identifying the problem, they will never be able to improve your solutions.

Smart companies resolve problems by analyzing them first and then building their solutions. How does your service provider measure up?

Want to learn more? Access a demo of our portal.

[1] There are of course some really great alien westerns out there, but they were built to be that from the beginning. Some examples would include The Mandalorian and Firefly but there are certainly many others. The point is that the intent behind why something is the way it is, is important.