Air-Gapped LLMs Still Fail RBI Audits When the Eval Harness Is Inside the Perimeter

The Architectural Confusion That Costs Banks Their Audit Sign-Off

On-premise LLM deployment compliance in BFSI India has become a boardroom topic, driven primarily by RBI's 2024 Master Direction on IT Risk and Cybersecurity, which tightened the conditions under which regulated entities can place critical workloads — including AI inference systems — with third-party cloud providers. The practical response at most Tier-1 private banks and large NBFCs has been to move LLM infrastructure behind the datacenter perimeter and declare the compliance problem solved. It is not solved. The air gap addresses exactly one regulatory obligation — data residency and localisation — and leaves the deeper assurance obligations structurally unaddressed.

The confusion is architectural, and it is understandable. Network isolation is visible, auditable in a single diagram, and maps cleanly to the language regulators use when they describe outsourcing risk and data sovereignty. Assurance integrity — the property that your evaluation signals accurately reflect model behaviour and have not been contaminated by the environment under test — is harder to draw on a diagram. But it is precisely what an RBI IT examination, a DPDP Act audit, and an EU AI Act conformity assessment are each, in their own language, asking you to demonstrate. A perimeter that keeps data in-country does nothing to prevent your eval harness from sharing runtime state with your inference stack, your production secrets from being accessible to evaluation workers, or your model weights from arriving in production without a verifiable chain of custody.

What RBI's 2024 Master Direction Actually Requires

RBI's Master Direction on IT Risk and Cybersecurity 2024 is not primarily a data-residency directive. It is a governance and control directive that happens to include data-localisation clauses. The sections most relevant to AI deployment concern IT infrastructure risk management, change management controls, and the auditability of third-party and outsourced technology. When an AI inference system is deployed on-premise, it does not automatically exit the scope of these obligations — it changes the nature of the entity responsible for meeting them. The bank assumes full ownership of every control layer: the model, the runtime, the evaluation pipeline, and the configuration management that holds them together. RBI examiners will look for evidence that each of these layers is independently governed and that the governance artefacts are traceable. A network diagram showing no cloud egress satisfies none of this.

The DPDP Act 2023 adds a parallel obligation. Where LLMs process personal data — and in BFSI use cases involving credit decisioning, customer service, or fraud detection, they almost always do — the entity deploying the model is a data fiduciary with explicit obligations around purpose limitation, storage restraint, and the technical and organisational measures used to protect the data. An on-premise deployment does not reduce these obligations; it concentrates them. The data fiduciary cannot point to a cloud provider's compliance certifications. Every technical control must be owned, documented, and demonstrable.

Three Technical Controls That Define Assurance Integrity in Air-Gapped LLM Deployments

The following three controls are not optional hardening measures. They are the structural minimum for an on-premise LLM deployment to produce assurance signals that can withstand independent scrutiny.

The first is model signing in offline registries. Every model artifact — base weights, fine-tuned checkpoints, adapter layers, tokeniser configurations — must be cryptographically signed before it enters the internal registry, and the signature must be verified at every subsequent stage: packaging, staging, and production promotion. The signing authority must be an internal PKI that has no certificate chain terminating in an external root that requires online validation. This control closes the supply-chain attack surface for model weights and creates the chain-of-custody evidence that a regulator can inspect. Without it, there is no verifiable answer to the question: is the model in production the model that was evaluated?

The second control is eval harness isolation. The evaluation pipeline — the harness that runs benchmark suites, behavioural tests, red-team probes, and drift detections against the model — must operate in a network segment that is not routable to the inference environment it is evaluating. This is not paranoia; it is basic measurement hygiene. If the eval harness can write to shared storage that the inference runtime reads, or if both share a service mesh, the harness is not measuring the model in isolation — it is measuring a system that includes its own measurement apparatus. In regulated AI, this contamination of assurance signals is a material governance failure. It means your evaluation reports do not reflect the model's behaviour under the conditions it will actually operate in. RBI examiners who understand model risk — and increasingly, they do — will ask how the independence of the evaluation environment is maintained. The answer must be structural, not procedural.

The third control is secrets management without internet egress. This is the most commonly bypassed air-gap control in practice. Credential managers and secrets vaults — HashiCorp Vault, CyberArk, and their equivalents — are typically configured during initial deployment to phone home for license validation, certificate revocation checks, or telemetry. In a genuine air-gapped environment, these calls either fail silently, fall back to insecure defaults, or trigger compensating behaviours that open unexpected channels. Every secrets rotation workflow, every dynamic credential issuance, and every audit-log forwarding path must be mapped and verified to resolve entirely within the internal network boundary. This requires active testing, not configuration review alone. The test is simple: capture all egress traffic during a full secrets rotation cycle and confirm that no packet leaves the internal boundary.

Regulatory Mapping: What Each Regime Is Actually Asking For

These three controls do not exist in isolation. They map to specific obligations across the regulatory instruments most relevant to BFSI AI deployment in India. RBI's Master Direction on IT Risk and Cybersecurity 2024 requires demonstrable change management controls and audit trails for IT systems — model signing and the offline registry directly satisfy this. It also requires that outsourced and third-party technology risk is managed with the same rigour as internal risk — eval harness isolation is the structural evidence that the evaluation function has not been outsourced informally to a shared runtime. The DPDP Act 2023 requires technical and organisational measures appropriate to the risk of processing — secrets management without egress is a direct technical control on unauthorised data access via credential compromise. The EU AI Act Article 10, which governs data governance for high-risk AI systems and is increasingly being referenced as a benchmark even by non-EU regulators, requires that training, validation, and testing datasets — and by extension, evaluation environments — are managed with documented governance and independence. Eval harness isolation is the infrastructure expression of that requirement.

Where Air-Gapping Alone Fails: The Continuous Assurance Problem

Physical isolation is a point-in-time control. A model that passed evaluation on the day of deployment begins to diverge from that baseline the moment it encounters production traffic, fine-tuning updates, or changes in the upstream data it depends on. Air-gapping does nothing to detect this drift. It does nothing to ensure that a model whose behaviour has shifted is caught before that shift reaches a credit decision, a fraud alert, or a customer communication that constitutes a regulated output.

Continuous assurance — the practice of running evaluation pipelines against production-equivalent model states on a defined cadence, with results that are independent of the deployment environment and traceable to a governance record — is the control that makes the air gap meaningful over time. Without it, the air gap is a snapshot of compliance at a single moment. With it, it becomes part of a living governance architecture that can answer the regulator's actual question: not whether the model was compliant when it was deployed, but whether it is compliant now, and how you know.

For BFSI organisations building or hardening on-premise LLM deployments, the sequence that survives examination is: establish the perimeter, then establish the assurance architecture that operates independently within it. The perimeter is necessary. It is not sufficient. Assurance integrity is the property that determines whether the perimeter holds any regulatory meaning at all.