Why AI Red Teaming for Financial Services Compliance Requires More Than a Pentest

What 'Coming of Age' Actually Means for Regulated AI

The CSOOnline piece on AI red teaming arriving at maturity is worth reading. It correctly identifies that adversarial testing of AI systems has moved from an informal practice inside a handful of large technology companies to something the broader industry is beginning to take seriously. That is a genuine milestone. But the piece is written for a security audience, and the maturity it describes — structured red team programmes, prompt injection libraries, jailbreak taxonomies — is largely a maturity in offensive technique. For a VP of AI Risk at a Tier-1 bank, technique is the least of the problem.

Maturity in regulated AI means something different. It means adversarial testing that generates evidence acceptable to a model risk committee, an internal audit function, and ultimately a regulator. EU AI Act Article 9 requires high-risk AI systems to have a risk management system that is continuous and iterative across the full system lifecycle — not a one-time assessment. The NIST AI RMF, in Measure 2.6, asks organisations to evaluate AI systems for trustworthiness properties on an ongoing basis. SR 11-7, the Federal Reserve and OCC guidance that governs model risk management in US banking, has long established that model validation must be independent, documented, and repeatable. None of these frameworks describe a pentest. They describe a discipline.

The gap between what Big Tech calls a red team programme and what a regulated financial institution needs is not a gap in ambition. It is a gap in governance architecture. Red teaming that lives inside a security team, produces a confidential report, and resets to zero at the next scheduled engagement is simply not the same artefact that model risk governance demands.

Why Episodic Red Teaming Fails Compliance Cycles

Consider the lifecycle of a credit decisioning model at a large bank. The model is trained, validated, approved by a model risk committee, and deployed. Over the following months, the input data distribution shifts as macroeconomic conditions change. The model is retrained on a new vintage of data. The prompt layer — if this is an LLM-assisted decisioning component — is updated to reflect a new product. Each of these is a material change event. Under SR 11-7, each one may trigger a validation requirement. Under EU AI Act Article 9, each one is precisely the kind of lifecycle event the risk management system is meant to cover.

An annual red team exercise misses every one of those events. It produces findings against the model as it existed at a specific point in time, filed in a report that becomes stale the moment the model changes. Worse, it creates a false assurance posture. A bank that can point to a red team report from eight months ago has documentation. It does not have assurance. Regulators conducting supervisory reviews of AI systems are increasingly sophisticated enough to ask when adversarial testing last occurred relative to the most recent model change — and an eight-month-old report against a model that has been retrained twice since is not an answer that satisfies.

The organisational question many institutions are wrestling with — should red teaming sit with AI Risk or with the CISO? — is a symptom of this structural problem. When adversarial testing is framed as a security activity, it naturally gravitates toward the security function. But when it is recognised as a model risk evidence requirement, it belongs inside the model risk governance framework, with appropriate collaboration from security. The framing determines the governance, and the governance determines whether the output is usable at audit.

What Embedded Adversarial Assurance Looks Like in Practice

Embedding adversarial assurance into a model risk programme means treating red team activity as a repeatable, triggered, and documented process rather than a scheduled event. In practice this has three characteristics.

First, adversarial testing is tied to model change triggers. Every material change to a model — retraining, prompt layer update, integration change, context window expansion — initiates a defined scope of adversarial coverage. The scope is proportionate to the nature of the change, not fixed to a calendar quarter.

Second, findings are logged as model risk evidence. They enter the model inventory, are reviewed by the model risk committee or its delegate, and remain traceable. A finding that is remediated has a closed status. A finding that is accepted as residual risk has an explicit risk acceptance record. The artefact produced is not a security report; it is a model risk management artefact.

Third, the adversarial function is independent of the model development team. This is a basic principle from SR 11-7 that the security framing of red teaming frequently violates — when the team that built the system also runs the red team exercise, the independence that gives the finding its evidential weight is absent.

AI red teaming for financial services compliance is not a more rigorous version of what Big Tech does. It is a different discipline, governed by a different set of obligations, producing a different class of output. The CSOOnline milestone is real. But for regulated institutions, the coming of age has not yet arrived — and the institutions that recognise that earliest will be the ones whose AI risk programmes withstand scrutiny when the first EU AI Act supervisory reviews begin in earnest.

Building that programme requires assurance infrastructure that connects adversarial testing to model governance, not just to security operations. That is the gap Qapitol's AI assurance service is designed to close.