Non-Deterministic, Goal-Directed, Unauditable: The Three Properties That Break Traditional QA for BFSI Agents

Why Traditional QA Was Never Designed for This

Testing agentic AI systems in financial services is not a harder version of testing a REST API. It is a structurally different problem. Traditional functional QA rests on a foundational assumption: given input X, the system produces output Y, and a pass/fail verdict is meaningful and repeatable. That assumption breaks the moment the system under test is goal-directed, stateful across sessions, and capable of selecting its own sequence of tool calls to reach an objective. Agentic workflows in claims automation, fraud detection, and loan decisioning meet all three of those conditions. The failure is not in the QA team's diligence — it is in the method itself.

The distinction matters because regulated enterprises are not shipping chatbots. They are shipping systems that take consequential, sometimes irreversible actions: denying a loan application, flagging a transaction as fraudulent, or triggering a claims payout. When those actions are the downstream product of a multi-step agent reasoning over retrieved context, external tool calls, and accumulated session state, the surface area that can go wrong is orders of magnitude larger than any deterministic test suite can cover. And when a regulator — or an internal audit function responding to EU AI Act Article 9 obligations — asks for evidence of controlled behaviour, a green UAT report is not that evidence.

Failure Mode One: Goal Drift Across Multi-Step Reasoning Chains

Goal drift occurs when an agent pursues a subtask in a way that satisfies the immediate step but undermines the original objective or introduces unacceptable risk. In a loan decisioning agent, this might mean the system correctly retrieves a credit bureau file, correctly computes a debt-to-income ratio, and then takes a reasoning path through a sequence of retrieved policy fragments that produces an approval recommendation inconsistent with the lender's stated risk appetite — without any single step being obviously wrong. Each function call returns a plausible result. The aggregate trajectory violates policy.

Traditional QA has no mechanism to detect this. Unit tests validate individual tool calls. Integration tests validate that components connect. Neither validates the trajectory — the sequence of decisions and intermediate states across a full agent session. Trajectory-level evaluation, by contrast, scores the agent's path through a task against a set of behavioural assertions: did the agent consult the required policy documents before recommending approval? Did it stay within its authorised tool scope? Did the reasoning chain remain consistent with the declared objective? These are not pass/fail questions. They are scored, auditable observations about agent behaviour across a distribution of inputs.

Failure Mode Two: Tool-Call Hallucination and Privilege Escalation

Agentic systems in BFSI typically operate with access to external tools: database queries, API calls to core banking systems, document retrieval from policy stores. Tool-call hallucination is a well-documented failure pattern in which the agent invokes a tool with fabricated parameters, invokes a tool outside its authorised scope, or constructs a tool call that appears syntactically valid but carries semantically incorrect arguments. OWASP LLM Top 10 lists excessive agency and insecure tool integration as distinct risk items precisely because these failures are not edge cases — they are predictable under certain prompt conditions.

The compliance relevance is direct. A fraud-detection agent that calls a sanctions-screening API with a malformed entity identifier may return a clean result for a flagged counterparty. A claims agent that escalates its own database access permissions through a poorly constrained tool definition creates an insider-risk-equivalent exposure without any human actor. Neither failure will appear in a functional test suite that only checks whether the tool returned a response. Evaluation-based QE covers this through adversarial probing: injecting inputs designed to elicit out-of-scope tool calls, testing permission boundaries explicitly, and verifying that the agent's tool-use behaviour remains within the envelope defined by the system's risk classification.

Failure Mode Three: Session-State Corruption Across Long Contexts

Most UAT cycles test an agent in a single-session, single-scenario configuration. Production agentic workflows in insurance and banking frequently involve agents that carry context across multiple interactions — a claims adjudication agent that retains prior session data to build a longitudinal picture of a claimant, or a KYC agent that accumulates document-verification state across asynchronous customer interactions. Session-state corruption occurs when earlier context — whether injected adversarially or simply accumulated through normal operation — begins to distort later reasoning in ways that are not visible at the level of any individual response.

This is the failure mode that is most likely to survive UAT and appear only in production, often under the least convenient conditions. A single-session pass/fail test cannot expose it. Evaluation-based QE addresses it through multi-turn scenario libraries, longitudinal behavioural monitoring, and explicit testing of context boundary enforcement — verifying that the agent does not carry forward state it was not designed to retain.

Traditional Functional QA vs. Evaluation-Based Agentic QE

The structural differences between the two approaches are worth stating plainly.

Traditional functional QA operates on discrete inputs, produces binary verdicts, assumes deterministic outputs, tests at the unit and integration level, and is executed once at a gate. Its fitness criterion is correctness on the test set.

Evaluation-based agentic QE operates on distributions of inputs and scenarios, produces scored behavioural observations, explicitly accounts for non-determinism through repeated sampling and statistical aggregation, tests at the trajectory and session level, and is designed to run continuously. Its fitness criterion is defensible evidence of controlled behaviour across the operating envelope.

The practical implication: a QA team that has invested heavily in automated regression suites and CI/CD pipeline integration is not starting from zero — but those assets address a different problem. The evaluation layer needs to be built on top of, not instead of, existing functional testing.

The Compliance Hook: NIST AI RMF and EU AI Act Article 9

NIST AI RMF's GOVERN function requires organisations to establish policies and accountability structures for AI risk. Its MEASURE function requires that AI systems are evaluated against their intended purpose under varied and adversarial conditions, and that those evaluations are documented. A one-time UAT cycle satisfies neither requirement for a production agentic system operating in a regulated context. The MEASURE function specifically calls for ongoing monitoring of AI behaviour — a continuous evaluation posture, not a gate.

EU AI Act Article 9 imposes risk-management obligations on high-risk AI systems that include establishing, implementing, and maintaining a risk-management system throughout the system's lifecycle. Claims automation, fraud detection, and loan decisioning agents are plausibly classifiable as high-risk under Annex III of the Act. The risk-management documentation Article 9 requires — covering residual risks, testing methodologies, and evidence of controlled behaviour — cannot be produced by a functional QA process that generates binary verdicts against a fixed test set.

Firms that have shipped agentic workflows without this evaluation layer are not in a comfortable position. They hold UAT sign-off documents. They do not hold the evidence that Article 9 and NIST AI RMF require.

What a Defensible QE Rationale Looks Like

For a VP of Engineering or Head of AI/ML taking a QE rationale upward to a risk committee or a compliance function, the argument is not that agentic AI is too complex to test. The argument is that the existing test method is mismatched to the failure modes that matter, and that evaluation-based QE is the method that produces auditable evidence of controlled behaviour. That distinction — between testing for correctness on a fixed input set and evaluating for controlled behaviour across a distribution — is the difference between a UAT report and a compliance artefact. Regulators and internal audit functions are increasingly capable of drawing that line themselves. Getting ahead of it is a quality engineering decision, not a vendor procurement one.