AIBefore AI learns from your history, ask what it never recorded
Historical records show what the old process noticed, which is less than what customers needed. Before an AI system learns from them, we propose reviewing what's missing with the people who did the work, and taking a decision on each gap.
Suppose an AI project in service operations starts from something the company already has: years of tickets and case notes. The history is used to train a model or to give an assistant examples of how requests were handled, and then to test it before go-live.
Our view: historical data records what the old process noticed and wrote down. An AI system that learns from it treats what was rarely recorded as rare, and it can't learn what was never recorded at all. So before the history is used, the business should review what's missing from it, with the people who did the work. For each gap, the review should end in a decision: keep a human path, fix the data, turn an unwritten practice into a rule someone owns, or change how success is judged. Where none of these is quick, the system should handle less to begin with.
This matters most when the system sets priorities or routes requests. An assistant that only drafts replies for a person to check carries a smaller version of the same risk.
A good score only covers what was recorded
The usual check is accuracy. Part of the history is held back, the system runs on it, and its answers are compared with what actually happened. The test cases come from the same history, so they share its gaps. A system can do well on every case that was recorded and still mishandle the requests that never were. The score only covers the work the records could describe.
A heating service company and its tickets
Consider a hypothetical company that installs and services heating systems for homes and small apartment buildings. It wants an assistant that sets a priority for each incoming request and proposes the next step, so dispatchers can give their time to difficult cases. The plan is to use several years of tickets as examples of how requests were prioritised, and to test the assistant against them.
Before that, the dispatchers and technicians sit down with the project team and go through what the tickets don't contain. The output is a short list of gaps, each with the decision taken on it. Four gaps come up.
Requests that never became tickets. Some long-standing customers call their technician's mobile directly. Building managers email the person they know. These requests were handled, often well, but they left no ticket. In the history, those customers look as if they rarely need anything.
Needs the form couldn't hold. The ticket categories were written for equipment faults. "The heating works but one flat stays cold" or "I can't get into the customer portal to book a visit" end up under Other, where they're often closed or moved by hand. The assistant learns that Other is low priority.
Judgement never written down. In a cold spell, one dispatcher moves customers who live alone, or who have small children, to the front of the queue. She knows them. No field records why. The assistant learns the written rule and misses the practice that protected people when it mattered most.
Closed doesn't mean solved. A ticket closes when the technician leaves. If the heating fails again two days later, the customer's call opens a new ticket. The history counts the first visit as a success either way. Where the next step chosen back then was the wrong one, the assistant gets credit for copying it.
A standard data quality check wouldn't flag these gaps, because every field that exists is filled in.
A decision for each gap
For the heating company, each gap leads to a different decision.
Keep a human path. Requests that arrive through technicians' phones stay with people for now. The assistant doesn't see them, and nobody pretends it does. If the company later wants them in the system, they first need a way to be logged, agreed with the technicians. Only then do they become data.
Fix the data before using it. Someone who knows the work re-reads a sample of Other tickets and relabels them, and the categories change for new tickets. The assistant is set up and tested on the corrected sample.
Make it a rule with an owner. The cold-weather priority becomes an explicit rule that the service manager owns and can change. The assistant applies it, and dispatchers can see when it has. The rule should use only what customers chose to tell the company and agreed to have kept. Where that's missing, the case stays with a dispatcher.
Judge by whether it stayed fixed. The test checks whether the heating was still working days later, using repeat calls on the same address. If the company can't link those calls yet, linking them becomes part of the project.
Where a gap can't be closed quickly, the team can narrow the scope. The assistant starts with the request types the history describes well, while the rest stays with dispatchers.
Test what the history leaves out
Before go-live, the team writes test cases by hand for each gap: the flat that stays cold, the portal problem, the customer who lives alone during a cold spell, the repeat failure. Dispatchers write them, because they know what these requests look like in real life. The assistant is judged on these cases as well as on the historical score, and a good historical score doesn't make up for failing them.
The review is worth repeating after go-live. A running system creates gaps of its own. Customers may learn which words get a faster answer, and some may stop sending requests they expect the assistant to misunderstand.
Two objections, and a cost
Data teams already profile their data, and evaluation tools can check whether a model treats groups of people differently. So one objection is that the review adds nothing. Those checks work on what's in the dataset and on the attributes that were recorded. They can't see a channel that was never logged or a need the form had no place for. They're worth running once the review has shown what the data leaves out.
A second objection concerns privacy: looking for who's missing sounds like collecting more personal data. Three of the four gaps in the example are about how requests were logged and closed, and the fix is often a process decision, such as a human path or an explicit rule. Where a fix would mean recording something sensitive about people, that's a separate decision, to be taken with the people concerned and with whoever is responsible for data protection. Keeping those cases with a person remains an option.
The review also has a cost. It takes time from the people who know the work best, and it can make the first release smaller than planned. In return, the team knows which work the system has been tested on.
Where to start
Before anyone trains or tests anything, ask the people who handle the requests which requests reach them without leaving a record, and what they do that the system doesn't know about. Their answers become the first gaps on the list.