Before you trust an AI pilot, ask how much time the expert had

Picture a management committee at a mid-sized industrial company. Three AI proposals are on the table. Sales wants an assistant that drafts quotes for configured machines. Service wants an assistant that answers technicians' questions from manuals and past repair reports. Customer care wants an assistant that drafts replies to complaints and suggests a remedy. Each proposal comes with a sponsor and a supplier's estimate for the work. Together they fit within the year's spending, and the committee approves them all.

None of the proposals mentions Elena. She's the senior application engineer who has spent years deciding which configurations work and which repairs are worth doing. The quoting assistant needs her to define what a correct quote looks like. The service assistant needs her to say which old repair reports can be trusted. The complaints assistant needs her to decide which remedies it may suggest. She also still has her own job.

In the second month, the quoting assistant starts its pilot, a limited trial on real requests. Elena reviews its quotes in the evening and answers the service team's test questions between customer calls. The complaints work waits. Her reviews get shorter. After a few weeks, the committee hears that the quoting pilot went well and puts the assistant into everyday use. In the third month, its exceptions start coming back to her, just as the service pilot needs reviewing, and the complaints work is still waiting.

Our view: a pilot result is only as good as the review behind it, and the people who can do that review are few. Several proposals can need the same person at the same time, and a budget shows only money. We propose that every AI proposal carries an expert-time budget: the named people whose judgment it needs, and the days per month it asks of them, including after the system is in everyday use. The committee compares the proposals person by person before funding them. During each pilot, the expert reports the review time she actually had, next to the results.

Where the expert's time goes

In the example, each supplier's estimate covers the supplier's work, and Elena's hours appear nowhere. Her time goes into four stages of each project.

Setting the rules. Before anything is built, someone has to say what a correct answer looks like, which sources the system may use and which cases need a person. In this company nobody else can say it as reliably as Elena, and saying it well takes several conversations, because much of what she knows she has never had to write down.

Building test cases. These are real examples with the agreed right answer, including the cases that went wrong before. Elena chooses them and checks the answers.

Reviewing the pilot. Someone has to read what the assistant produced and say where it's wrong. This stage arrives when the project is already running and everyone expects it to be quick, so it's the easiest one to squeeze.

Living with the system. Once the assistant is in everyday use, exceptions come back to the person who knows the rules, and so do the adjustments when the business changes. In the plan, the project is already finished.

Illustrative example with a hypothetical company and days, titled "Three proposals, one expert". It shows what three approved AI projects ask of Elena, the senior application engineer, who has about four days a month for project work beyond her own job. Month 1: quoting assistant, set the rules and build test cases, 2 days; service assistant, say which old repair reports can be trusted, 2 days; 4 days asked, 4 available. Month 2, marked as asked for more than she has: quoting assistant, review the pilot quotes, 4 days; service assistant, answer the test questions, 3 days; complaints assistant, set the rules, waiting, 2 days; 9 days asked, 4 available. Month 3, also marked: quoting assistant, handle exceptions in everyday use, 2 days; service assistant, review the pilot answers, 4 days; complaints assistant, set the rules, still waiting, 2 days; 8 days asked, 4 available. A note says each proposal came with its own sponsor and supplier estimate, and none of them counted Elena's days. — Illustrative example: each project fits its own budget, but together they ask one expert for about twice the time she has.

What a squeezed review does

When there isn't enough time, the review gets shorter. Elena reads the first quotes carefully and skims the rest, approving what looks plausible. The pilot report says a person reviewed every quote. The errors she didn't have time to find reach customers later, when they cost more and are harder to trace back.

It's also unfair on her. The projects were meant to make her work easier. They added a second job, and if a problem surfaces later, her approval is on the record.

The expert-time budget

For each proposal, the budget names the people whose judgment the assistant needs and estimates their days per month at each of the four stages. A rough estimate is enough to make the demand visible. It's prepared with those people, since they're best placed to say how long setting the rules will take. The sponsor owns it and brings it to the committee with the proposal.

The committee then puts the proposals side by side, person by person and month by month. Where someone is asked for more days than they have, it has to choose: run the projects one after another, prepare a second person to take part of the work, or take something off that person's current job and say who covers it. Each option has a cost the committee can see. The comparison also shows the other people named in the budgets, so the committee can see who could take on some of it.

In the example, the committee might start with the complaints assistant, because a wrong remedy does the most damage and its review deserves Elena's full attention. Or it might start with the quoting assistant and give the service assistant's test questions and pilot review to a second engineer, with Elena checking a sample of that engineer's work.

Illustrative example with a hypothetical company and days, titled "Compare by person, month by month": what the committee would have seen before funding, with the expert-time budgets of the three proposals side by side, in days per month. Elena, senior application engineer: month 1, 4 days asked, 4 available, for quoting and service; month 2, 9 days asked, 4 available, for quoting, service and complaints, highlighted in red; month 3, 8 days asked, 4 available, for quoting, service and complaints, highlighted in red. Head of service: 1, 2 and 2 days asked, 3 available each month, for service. Customer care lead: 0, 1 and 1 days asked, 3 available each month, for complaints. A navy box headed Before funding says: where someone is asked for more days than they have, choose one project after another, a second person for part of the work, or relief from part of their current job. A navy box headed During each pilot says: the expert notes the review time she actually had; below the agreed minimum, only reviewed outputs go out, or the pilot pauses; nobody calls it a success until she has had that time. — Illustrative example: the same budgets, compared person by person, show the overload before anyone approves the projects.

Review time as part of the result

The budget also sets a minimum review time for each pilot, agreed between the sponsor and the expert before it starts. During the pilot, the expert notes each week how much time she actually had for review. She writes it herself, in a few lines. The pilot results go to the committee with that note and against the success criteria agreed at the start.

If the review time falls below the minimum, the pilot continues only on outputs someone has reviewed before they go out, or it pauses. Nobody calls it a success or extends it until the expert has had the agreed minimum time for review. The committee takes that decision, based on the expert's note.

Objections and limits

The first objection is that this is ordinary resource planning. Even where a resource plan names people, the time spent judging AI output is easy to miss. It peaks at the pilot review, which the plan expects to be quick, and it continues once the system is in everyday use, after the project has left the plan.

The second objection is that AI is meant to save the experts' time. Once an assistant is working well, it may. To get there, it takes their time first, and the company should plan for that period instead of discovering it.

The budget matters most where the result depends on someone's judgment. If an assistant's output can be checked automatically, for example against an order already recorded in the company's systems, the expert's review time matters less. The cost of the approach is that fewer projects run in parallel and some start later.

Where to start

Take the AI proposals currently waiting for approval and, for each one, write down the names of the people who will have to say what a right answer is. If the same name appears more than once, talk to that person before the committee meets.