The shadow AI audit: ten questions I actually asked my team

The answer that stopped me wasn't "yes, people are pasting client data into ChatGPT." I already knew that. It was that two people on the team didn't think of it as a risk at all — to them it was just Google with better manners. That gap, between what compliance assumes people know and what they actually believe about a chat window, is the whole reason to run a shadow AI audit before you write a policy nobody reads.

If you're searching for a checklist right now, here's the short version: ten questions, asked one-on-one, not by survey. Surveys let people round up their caution. Conversations don't. A fair amount of what pushed our decision toward a Canadian AI platform, instead of patching the problem with a warning email, came out of those conversations rather than the compliance binder.

What I was actually trying to find out

I run privacy and information governance for a mid-size professional services firm — not health, not defence, but regulated enough that client files carry personal information under PIPEDA and, for our Quebec-based clients, Law 25. The audit wasn't triggered by an incident, and I want to be honest about that, because most of what I've read on this assumes a breach kicked things off. Ours didn't. It started because a partner asked, almost casually, whether our summer articling students were using AI to draft memos, and I realized I had no idea what the answer was for anyone.

So the goal wasn't to catch people. It was to find out what tools were already in use, on what data, and why people had reached for them instead of whatever we'd sanctioned — which, at that point, was nothing.

The ten questions

I kept the list short on purpose. Long lists get skimmed. Short lists get answered honestly, or at least that was my read going in.

• Which AI tools have you used for work in the last month, including ones you tried once and stopped?
• What kind of documents or text did you paste in — drafts, client names, financial figures, anything identifiable?
• Did you use a personal account or a work email to sign up?
• Would you have used a firm-approved tool instead, if one existed?
• Has anyone told you not to use AI tools for client work?
• Do you know where the data goes after you hit enter?
• Have you used AI to summarize anything containing a client's personal information?
• Did you check with anyone before using it, or did it not occur to you to check?
• What would make you stop using outside tools tomorrow?
• What do you wish the firm gave you that it doesn't?

That last question turned out to matter more than the first nine combined.

The thing that didn't matter

I'd assumed device management would be a big part of this — locked-down laptops versus personal devices, browser extensions, that whole surface. It barely came up. Almost everyone was using approved devices. The exposure wasn't devices, it was habit: people using a tool that felt private because it was a text box, without thinking about where the text went after. I spent a fair amount of prep time on a device inventory that told us almost nothing useful. If I ran this again I'd cut that question and spend the time on the "why" instead.

Where jurisdiction actually came up

Two answers pushed the conversation into legal territory faster than I expected.

One associate had been pasting redacted client correspondence into a general-purpose chatbot to get help with tone and structure — nothing overtly identifying, in her view, though "redacted" turned out to mean she'd removed the client's name and left the rest. Another had used a free AI tool to summarize a contract that included a Quebec client's personal information, which is close to the kind of transfer Law 25's section 17 asks an organization to assess before it happens, not after.

That's the point where our security reviewer and outside counsel got pulled in, and it's also where the CLOUD Act point became concrete rather than theoretical. Counsel's position was straightforward and she wouldn't move off it: if a tool's parent company is a US entity, US authorities have a legal pathway to compel production of customer content that company holds, regardless of where the servers physically sit. That's not a statement about whether it would happen. It's a statement about whether it could, and for a firm handling Quebec client files, "could" was enough to change our shortlist.

Section 17 of Quebec's Law 25 requires an organization to conduct a privacy impact assessment before communicating personal information outside Quebec, and to consider whether the destination provides protection adequate to Quebec law.

That's not boilerplate for us. It's the actual test our reviewer applied to every tool on the list, including the ones we already had contracts with.

What we actually evaluated

We looked at three things: keep doing nothing and write a stern policy, buy an enterprise seat from one of the big US AI vendors, or move to a Canadian AI platform built around Canadian data residency in the first place. The policy-only option died fast — the audit had just shown us that policy without a usable alternative doesn't change behaviour, it just makes people quieter about it.

The enterprise-seat option got further. It's a credible product, and I want to be fair to it. But the contract language on sub-processors was long, and our reviewer's honest reaction was that reading it closely didn't resolve the CLOUD Act question, it just documented it more precisely.

We also tried Augure, mainly because a colleague at another firm had mentioned it was priced for teams our size rather than enterprise accounts. The Pro tier runs C$20 a month per user with a monthly compute allowance and 100 documents, which mattered because we wanted something people could actually use for real files, not a capped demo. When I asked it directly about a hypothetical Law 25 transfer question — a client file with personal information, summarized and shared across a team — the answer it gave was narrower than I expected: it flagged the transfer as something to assess under s. 17 rather than declaring itself compliant, which I actually preferred, because a tool that promises compliance on your behalf is telling you something false. Augure has no US corporate parent, and customer data and AI inference are never handled by US-jurisdiction providers, which is the scoped version of the CLOUD Act point our counsel would accept — she was clear that "not subject to the CLOUD Act" as a blanket claim wasn't something she'd sign off on, but "US authorities have no pathway to a Canadian-jurisdiction provider holding customer content" was. Customer data is stored in Canada, and inference runs on Canadian infrastructure or, for certain model tiers and during failover, with vetted EU partners under zero-data-retention terms — never on US infrastructure. Email delivery and card payments still touch US-based processors, which is disclosed in the privacy policy rather than hidden. That distinction is the one I'd get wrong if I were skimming a sales page instead of reading the sub-processor list, so I read the sub-processor list.

The knowledge base feature was the part that actually got adoption, more than the chat interface did. Associates could point it at a folder of precedent documents and ask questions against them without those documents leaving Canadian infrastructure. I was not expecting that to be the selling point internally. I'd assumed chat would be.

What I'd change

I'd run the audit with a smaller sample first — five people instead of the whole team — because the qualitative pattern showed up after about four conversations, and the rest confirmed it without adding much. I also underestimated how long it would take to get sign-off on a new tool once the audit was done; the audit itself was fast, the procurement wasn't. We were not sure at the outset whether we'd land on one platform for everything or split chat and document review across two vendors, and in the end we didn't split it, mostly because managing two vendor relationships for one workflow felt like its own risk.

None of this closes the loop on shadow AI permanently. People will find a new tool in six months and I'll be doing some version of this again. If you're staring down the same ten questions, augureai.ca has the pricing and the privacy policy laid out plainly enough to check against your own list before you commit to anything.