AITest your AI assistant as someone who shouldn't see the answer
Explore the perspectiveAn internal AI assistant can hand someone a sentence from a document they've never been able to open. Before it goes to everyone, we propose testing it as the people who should be refused, with the owners of the content deciding what a correct refusal is.
In this article ＋
Picture a mid-sized engineering company opening an internal AI assistant to all staff. It answers questions from procedures, contracts, HR policies and project folders. In the first week, Marta, a project coordinator, asks what's planned for the logistics team next year. The answer is clear and well written. One sentence comes from a draft reorganisation plan that sits in the leadership folder. Marta has never been able to open that folder.
Marta signed in with her own account. The assistant searched only the index it had been given. That index had been built with a technical account, an account used by software rather than by a person, which could read every folder because the project team wanted complete answers. The leadership folder's permissions were never changed. Nothing between the folder and Marta's screen checked them again.
Our view: before an internal assistant goes to everyone, the test that matters most is what it refuses, and to whom. We propose testing it with accounts that have the same access as real roles, including the roles that should be refused, against questions written for the purpose. The owners of each kind of content decide in advance what a correct answer and a correct refusal look like. The release goes ahead when they and the security lead have signed off on the results.
Where the question "who's asking?" gets lost
Permissions live in the systems where documents are kept. A file server, a contract archive or an HR system knows who may open what. To make an assistant useful, content usually leaves those systems. It gets copied into a search index, a prepared copy of the content that the assistant searches instead of the original folders. Along the way it may be split into passages, summarised, stored next to other content and passed between tools.
Each of those steps is a new place where the original rule has to be applied again. Each handover between tools is a point where the request can stop carrying who asked and start carrying a technical account instead. Each tool the assistant connects to adds a handover.
What was shared too widely becomes easy to find
Some content may already be shared wider than intended before any assistant arrives. In a company that has used shared folders for years, it might be a spreadsheet of salaries saved in a project folder, or a contract draft in a space open to the whole department. Until now, finding files like these meant knowing where to look.
An assistant that respects permissions perfectly will still find them, for anyone who has access, on the first question that touches the subject. What used to be accessible but hard to find is now one question away. The fault belongs to the source, but the assistant makes it visible, and the test can find it before staff do.
The refusal test
Draw the chain. For one assistant, list every place content is copied, indexed or stored, and for each step write down whose account is used. IT can prepare it, and for one assistant it often fits on a page.
Ask the content owners. The people responsible for each area, such as HR, finance, legal or a project lead, say who may know what, in business terms. Salary information might be for HR and the person's own manager. A reorganisation draft might be for the leadership team until it's announced. IT applies these rules to the chain and, where needed, to the source. It shouldn't have to guess them.
Write the questions and the expected results. Content owners and IT write a fixed set of questions together: ordinary ones, questions about sensitive subjects, and indirect ones, such as asking for a summary of the changes planned for a team. These are questions real employees would ask. The test isn't an attempt to trick the assistant, and it doesn't replace security testing.
IT creates dedicated test accounts with the same access as each role, not the accounts of real people. For each question and each role, the owners write the expected result before anything runs: an answer, a partial answer or a refusal. A partial answer gives only what has already been announced.
Judge the refusals strictly. A refusal passes only if it doesn't reveal the content another way. A summary, a quoted document title or a suspiciously specific "I can't share the details of the plan for your team" all count as failures, because each one confirms that something exists.
Two kinds of failure, two owners
When the test finds an answer that shouldn't have been given, the first question is where it came from. If the person couldn't have opened the source document, the chain is at fault: the index, the technical account or a stored copy. IT fixes that before release.
If the person could already open the source, the source is at fault, and the owner of that content area decides whether to tighten its permissions. Until they do, IT can remove that area from what the assistant searches. In both cases someone is named and the finding is closed before the assistant goes to everyone.
The release is signed by the content owners and the security lead together, on the basis of the test results. Their signature means they've seen what the assistant gives and refuses to each role, in business terms rather than in a technical log.
When to run it again
The test isn't done once. The chain changes with new sources, reorganised folders and a rebuilt index, and people change roles. The same questions and accounts can be rerun after each change, and the content owners only need to look at the differences.
Objections and limits
The first objection is that enterprise AI platforms already apply the permissions of the systems they connect to. Where a platform reads a source directly and checks the person's own access, that part of the chain is covered. The risk sits in what the company builds around it: an exported folder, a custom index, summaries saved somewhere else, a shared technical account for an automation. It also sits in permissions that were already too wide, which a platform has no way to judge, because only the content owner knows who should see what.
The second objection is that the company should fix permissions at the source first. It should. Without a test by role, though, it doesn't know where to look, and a full review of every folder is a long job. The test shows where the problems are that the assistant can actually reach.
The approach has a cost. Someone has to create and maintain the test accounts, the content owners have to give time to writing the rules and reviewing the results, and some content will be left out of the first release. Where an assistant only uses content that's public or open to everyone in the company, a light version is enough: check that the chain doesn't pull in anything else. This test covers what an assistant tells people. Assistants that take actions in other systems need further checks.
Where to start
Take the assistant closest to going live. Ask HR and the leadership team which subjects should never reach a general employee, then sign in with a test account for that role and ask about them.
Keep exploring.
AIWhen an AI number improves, check what changed underneath it
An AI workflow runs on services that suppliers update on their own schedule, sometimes including how use is counted. We propose a company change log covering your changes and theirs, and fixed test requests rerun after every change, before anyone reads the trend.
Read more
AIBefore you trust an AI pilot, ask how much time the expert had
A pilot result is only as good as the review behind it, and the people who can review it are few and can be wanted by several projects at once. We propose budgeting their time before funding and comparing proposals person by person. Every pilot result then comes with the review time the expert actually had.
Read more