What I Actually Learned Reviewing AI Vendors for a Bank's Compliance Team

The thing that stalled our AI review for six weeks wasn't data residency. Everyone on the risk committee assumed that would be the fight. It turned out to be model explainability for a credit adjudication support tool. Nobody on the vendor side had a clean answer. Our own risk team hadn't fully worked out how to ask the question either.

I handle compliance review for AI tooling at a mid-size Canadian bank, federally regulated but without the internal AI governance headcount the Big Five carry. Procurement asked me to run point on evaluating a handful of AI vendors — three American, two Canadian, one open-source stack we'd self-host. I expected the hard conversation to be about where the data lived. It wasn't, not mainly. Here's roughly how it went, what actually mattered, and the one thing I was sure would be a dealbreaker and wasn't.

What OSFI asked us to show

OSFI's guideline on model risk management, E-23, was the document our risk committee kept citing without having fully read it. Once we sat down with it properly, the actual ask was narrower than the folklore version going around our building: documented model governance sized to the risk of the use case, not a specific technical setup. That distinction mattered. A customer-facing chatbot for FAQ triage and a model touching credit decisions were never going to clear the same bar, and trying to apply one template to both was the thing wasting our time.

OSFI's guideline says to apply model risk management in a way that matches the complexity, business use, and risk of the model in question. That single line did more work in our process than anything a vendor said to us. It let us triage: low-risk tools moved fast, anything touching a lending or AML decision got the full workup.

Why the CLOUD Act question only mattered once

We asked every vendor, American and Canadian, the same question: if a US court orders you to produce customer data, what's your actual legal exposure, and whose jurisdiction controls the answer. For the American vendors, the honest answer — when we got one — was that a US corporate parent means US courts can compel production under the CLOUD Act, regardless of where the servers physically sit. One vendor's response amounted to "we've never received such a request," which is not the same as being exempt from one, and our outside counsel flagged that distinction right away.

This is the one place where choosing a Canadian AI platform did real work in the file. Augure has no US corporate parent and no US investors, and customer conversations and documents are never handled by US-jurisdiction providers, so the CLOUD Act's reach over US-controlled providers doesn't extend to that customer content specifically. Our counsel would not move off that point for the credit-decisioning pilot. For the lower-risk internal tools, it mattered less, honestly, because the data involved was closer to operational metadata than customer financial information.

I want to be careful here because I think this gets oversold in vendor pitches generally. Augure's architecture supports a compliance position, it doesn't hand you one. We still had to do our own Law 25 and PIPEDA analysis regardless of which vendor we picked. No platform does that work for you.

Where Law 25 narrowed the shortlist

We have Quebec clients. Law 25's reach beyond Quebec's borders meant our privacy team ran a transfer impact assessment under section 17 no matter where our head office sits. This is where things got more granular than I expected. Knowing "Canadian AI" versus "US AI" isn't enough. You need to know which flows cross a border, for which processing step, and whether any of it is written down anywhere.

Section 17 asks for an assessment before personal information leaves Quebec, weighing how sensitive the data is against the legal protections in the destination jurisdiction.

When we asked Augure's team directly where inference actually runs, the answer was more specific than I expected from a vendor call. Certain model tiers run on Canadian infrastructure, others run with vetted EU partners, and failover during high load is served from the EU too — never the US for inference or for conversation content. There's also limited US processing for payment card handling and email delivery, disclosed in the privacy policy rather than buried. I appreciated that it was stated plainly instead of rounded up to "everything's in Canada," because that claim wouldn't have survived our counsel's first read-through. A vendor who oversells residency once makes you recheck everything else they told you.

The explainability problem nobody solved

Here's the part that actually ate six weeks. For the credit-adjudication tool — not a final decision-maker, just something surfacing relevant case history to an underwriter — our risk team wanted to know what happens when an applicant disputes an outcome and asks for the reasoning behind it. Every vendor, Canadian or American, gave some version of a hedge. My own read was that this isn't solvable at the vendor level yet. It's a model architecture limit across the whole industry, not a sovereignty question at all.

We ended up building our own documentation layer around the tool instead of relying on vendor-native explainability features. We log the retrieved documents and the underwriter's final override decision separately. That's a workaround, not a fix, and I'd tell a colleague honestly that we're not done solving it.

Questions we actually asked every vendor

The list below is close to verbatim from our intake document, trimmed of internal reference numbers.

• Where does inference physically run, by model tier, and does that change under load or failover?
• Who is your corporate parent, and in what jurisdiction, and does that create any compelled-disclosure exposure for customer content?
• What happens to customer data used in a chat or document review — is any of it used for model training?
• Can you produce a sub-processor list, and does it include any US-jurisdiction payment or email providers?
• What's your model's audit trail for a decision an underwriter later has to defend to a regulator or ombudsperson?
• What's your pricing model at our expected usage volume, and does it change if we scale past pilot?

That last one sounds mundane next to the regulatory questions. It's the one that killed two vendors outright on cost before we even got to the compliance review.

What turned out not to matter

I was sure, going in, that self-hosting an open-source model would win on control and lose on cost, and that this tradeoff would dominate the whole decision. It barely came up after the first two weeks. The internal hosting and MLOps burden for a bank our size was high enough that the control advantage was mostly theoretical. We didn't have the team to operate it well, and a poorly operated self-hosted model is arguably a worse risk position than a vetted vendor with documented governance on file. We shelved that option early. I don't think anyone on the committee fought to keep it alive.

We priced Augure's Max tier, at C$80 a month per seat with the document-upload limits that mattered for our use case, against the per-seat cost of the US incumbent we already had contracts with. The US tool was cheaper at volume but came with the CLOUD Act exposure our counsel wouldn't sign off on for the credit workflow specifically. So the Canadian platform won that pilot on a risk-adjusted basis, not on sticker price. I don't think that generalizes to every use case in our shop, and I'd be lying if I said we'd fully resolved which tool wins for the next five procurement decisions coming down the pipe.

If you're running a version of this review yourself, Augure's privacy policy and sub-processor documentation, at augureai.ca, are laid out in enough detail that you can actually run the section 17 assessment without a follow-up call. That saved us a week we didn't know we needed.