When an AI number improves, check what changed underneath it

Picture a monthly review a few months after an AI assistant went live in customer service. The assistant drafts replies to customer requests that arrive by email, and it closes the simple ones on its own. Two lines on the dashboard look good: a larger share of requests is closed without anyone touching them, and the cost per request has gone down. The proposal on the table is to extend the assistant to a second service team.

Two suppliers sit behind the assistant. The model supplier provides the AI model that writes the replies. The platform supplier runs the assistant and produces the usage report the dashboard is built on. They can be two different companies, and each one updates its service on its own schedule.

In the same weeks, three things changed. The service team rewrote the standard closing reply the assistant sends. The company had chosen to always use the model supplier's latest version, so when a new version came out, the assistant switched to it overnight. And the platform supplier changed what its usage report counts: requests that ended in an error are no longer included unless someone asks for them. Only the first change was the company's decision. The other two arrived as release notes in an IT inbox.

Nobody in the room can say which change moved the numbers. The dashboard takes its total from the usage report, and requests that ended in an error were never closed. Removing them shrinks the total, so the share closed automatically rises with no change in the work. Cost per request moves for more than one reason. Fewer requests in the count would push it up, unless the cost figure dropped the same failures. The new model may charge less, or write shorter replies that also say less.

Our view: when a number moves after go-live, the first question is what changed underneath it. Some changes alter the work the assistant does. Others alter only how the work is measured, and those can move a number just as far. Good news deserves this question more than bad news does. We propose that the company keeps its own change log for each AI workflow, covering its changes and its suppliers', with a named owner. After every entry, a fixed set of test requests runs again. Each review of the numbers starts from the log and those results.

When a key number drops, someone investigates. They open the error records, ask the service team what it changed, and often find the supplier's notice. When a number rises, it goes into the steering committee slides and becomes the argument for the next investment. If the rise came from a change in what's counted, the company extends the assistant on a result that wouldn't have appeared under last month's definition. The second service team then inherits a target the first one never reached, and its people are judged against a number that was never real.

In the example, the people who read supplier notices and the people who read the dashboard sit in different departments. IT sees the notice about the usage report and files it as a reporting detail. The operations director sees the dashboard and has no reason to go looking for a notice. No one was asked to put the notice next to the dashboard.

A new model version comes with notes about what it does better in general. How it handles your requests, with your closing reply and your business rules, is something only your own tests can show.

The owner follows the notices from every supplier the workflow depends on, decides which ones matter and writes them in. The role suits someone who can read a technical notice and knows which numbers the business decides on. The role needs a deputy and scheduled time, or the log goes quiet the first busy month.

To see what a change did to the work, the service team keeps a fixed set of test requests: real requests, anonymised where needed, covering everyday work and the cases that went wrong before. Each one comes with the reply the business has agreed is right. After any entry in the log, on either side, the set runs again and the results are compared with the previous run. A change can take a little while to reach everything, so the rerun waits until it has fully taken effect.

If the model changed and the test requests come out the same, the rise on the dashboard needs another explanation. If they come out differently, the service team knows where to look before customers notice.

Where a supplier lets the company stay on one model version, staying is sensible for anything that feeds a business decision. Moving to a new version then happens deliberately, once the test requests have run on it. A fixed version stays available only as long as the supplier keeps it, so the move still has to be planned.

IT already runs change management, and some suppliers let the company stay on a fixed version. Change management covers the changes the company asks for. Supplier changes arrive as notices, often to defaults or to reporting, and not every part of a service can be held still. The change log sits alongside change management and adds the changes nobody requested, each linked to the numbers it can move.

A second objection is that monitoring tools already do this. Monitoring measures with the definitions it's given. When a definition changes upstream, the dashboard stays consistent with itself and stops being comparable with the month before.

The business has to agree the test requests once and keep them current, and someone who understands both the supplier notices and the dashboard has to give the log regular time. For an assistant that only drafts text for a person to check, a lighter setup is enough. The full log and test set are worth the effort where the numbers decide whether the system grows.

List the external services your AI workflow depends on and, for each one, find where its supplier announces changes. Then ask who read the last notice, and whether anyone checked it against the numbers your leadership looks at.