Synthetic Data Quality Scorecard

Why Synthetic Data Needs a Quality Gate

Synthetic data has become a practical necessity in enterprise AI assurance. Source datasets carry PII, are too small to stress-test edge cases, or are locked behind data-sharing agreements that take months to clear. Synthetic generation sidesteps many of those constraints. But it introduces its own class of failures — and those failures are quiet. A synthetic dataset can pass a cursory visual inspection and still produce a model that systematically underperforms on a minority class, violates a regulatory business rule, or leaks partial information about real individuals.

In regulated environments — banking, insurance, healthcare — the consequences of that silence are material. A synthetic training set that misrepresents the distribution of a protected attribute produces a biased model. A synthetic test set that omits valid edge-case transactions produces false test coverage. Neither failure is visible until something downstream goes wrong.

This scorecard gives you a structured, repeatable gate. It is organized into four quality dimensions. Each dimension contains scored criteria, failure indicators, and guidance on evidence. The scorecard is designed to produce an auditable record — one you can reference in an ISO/IEC 42001 AI management system review, a NIST AI RMF governance checkpoint, or an EU AI Act conformity assessment for a high-risk AI system.

Score each criterion on a 0–3 scale: 0 = criterion not met or not assessed, 1 = partially met with known gaps, 2 = met with minor reservations, 3 = fully met with documented evidence. Aggregate within each dimension and apply the dimension weights appropriate for your context (defaults given below). A dataset that scores below the minimum threshold on any single dimension should not proceed, regardless of its total score.

---

Dimension 1 — Distribution Fidelity

Default weight: 30% of total score.

Distribution fidelity measures how accurately the synthetic dataset reproduces the statistical structure of the real source data. A high-fidelity synthetic dataset is not a copy of real data — it is a population that behaves like the real population at the feature, joint-distribution, and temporal levels.

Criteria

• ▪Marginal distribution match: For every feature, compare the synthetic distribution to the source using an appropriate divergence metric (Jensen-Shannon divergence for categorical features; Kolmogorov-Smirnov or Wasserstein distance for continuous features). Score 3 if all features pass your threshold; score 1–2 if a minority of low-importance features show moderate drift.

• ▪Pairwise correlation preservation: Compute the correlation or mutual information matrix for the source and compare it to the synthetic equivalent. Pay particular attention to pairs that carry business meaning (e.g., age and product eligibility, diagnosis code and procedure code). A synthetic dataset that breaks real correlations will misdirect any model trained or tested on it.

• ▪Tail and rare-event coverage: Verify that the synthetic dataset preserves — or deliberately amplifies, if that is the stated purpose — the frequency and character of rare events. For fraud detection, this means rare positive labels. For clinical risk scoring, it means high-acuity episodes. Document the source rate and the synthetic rate explicitly.

• ▪Temporal and sequential integrity: If the data has a time dimension (transaction sequences, claim lifecycles, patient journeys), validate that temporal ordering, seasonality, and lag relationships are preserved. A synthetic time-series that shuffles event order produces nonsensical sequence features.

• ▪Class and label balance documentation: Record whether the generation process deliberately altered class balance (e.g., oversampling fraud cases for test design). Any deliberate imbalance must be flagged so downstream consumers do not mistake it for population truth.

Minimum pass threshold for Dimension 1: Score 2 or above on marginal distribution match and pairwise correlation preservation. A score of 0 on either is an automatic fail for the dataset.

---

Dimension 2 — Business-Rule Conformance

Default weight: 25% of total score.

Statistical fidelity is necessary but not sufficient. Synthetic records that are statistically plausible but violate business rules produce test suites with structural blind spots and models that learn impossible patterns as valid signal.

Criteria

• ▪Referential and relational integrity: In multi-table synthetic datasets, foreign key relationships must be preserved. A synthetic customer record must link only to synthetic accounts that exist in the synthetic account table. Orphan records or impossible joins are disqualifying defects.

• ▪Domain constraint satisfaction: Every field-level business constraint that exists in the real system must be encoded and enforced in the synthetic generator. Examples include: date-of-birth must precede policy-inception date; loan amount must be positive and within product tier limits; ICD-10 code must belong to a valid code set. Enumerate your constraints in advance and test each one programmatically.

• ▪Cross-field logical consistency: Check rules that span multiple fields. A synthetic health record with a pediatric age and an adult-onset-only diagnosis code is internally inconsistent. A synthetic trade record with a settlement date before the trade date is invalid. These are not statistical anomalies — they are categorical errors that real data would never contain.

• ▪Regulatory classification accuracy: In BFSI and insurance contexts, verify that synthetic records are classified correctly under applicable regulatory categories (e.g., product classification, risk band, reporting threshold). A synthetic dataset used to test a model that drives regulatory reporting must conform to the classification logic of the relevant framework.

• ▪Null and missing-value patterns: Real data has structured missingness — some fields are blank for specific record types or customer segments. Synthetic generators often produce uniform missingness or uniform completeness, both of which distort model behavior. Validate that null patterns match the real schema.

Minimum pass threshold for Dimension 2: Zero tolerance on referential integrity failures and domain constraint violations. Any confirmed violation is an automatic fail. Score the remaining criteria on the 0–3 scale.

---

Dimension 3 — Privacy Guarantees

Default weight: 25% of total score.

Synthetic data is not automatically private. Depending on the generation method, a synthetic dataset can re-identify individuals through membership inference, attribute inference, or linkage attacks. In jurisdictions covered by the EU AI Act, India's DPDP Act, or sector-specific rules (HIPAA, GDPR), the assumption that synthetic equals anonymized is not defensible without evidence.

Criteria

• ▪Membership inference resistance: Assess the probability that an adversary with access to the synthetic dataset can determine whether a specific real individual was in the training population. For generative models trained on sensitive data, this requires formal or empirical membership inference evaluation. Document the attack model and the result.

• ▪Attribute inference resistance: Evaluate whether an adversary can infer a sensitive attribute of a real individual (e.g., health condition, income band) by querying the synthetic dataset. This is distinct from membership inference and requires separate evaluation.

• ▪Nearest-neighbor distance and outlier proximity: Compute the distance between each synthetic record and its nearest real record in the source dataset. Flag synthetic records that are very close to a unique real record — these are the highest re-identification risk. A formal metric here is the Distance to Closest Record (DCR) distribution; the proportion of synthetic records with DCR below a defined threshold should be documented and minimized.

• ▪Differential privacy accounting (if applicable): If the generation process incorporates differential privacy, document the privacy budget (epsilon value), the noise mechanism used, and the composition across queries or training runs. An epsilon value without mechanism context is not an adequate privacy disclosure.

• ▪Regulatory sufficiency assessment: Record a documented opinion — not an assumption — on whether the synthetic dataset qualifies as anonymized or pseudonymized under the relevant regulatory framework for the intended use. This opinion should be revisited if the use case changes.

Minimum pass threshold for Dimension 3: A score of 0 on membership inference resistance or attribute inference resistance is an automatic fail. Datasets intended for external sharing require documented evidence, not estimates.

---

Dimension 4 — Downstream Test Effectiveness

Default weight: 20% of total score.

The final dimension asks the most practical question: does the synthetic data actually do the job it was created for? A synthetic dataset that is statistically faithful, rule-conformant, and privacy-safe can still be ineffective if it fails to surface the failure modes the test suite was designed to catch.

Criteria

• ▪Test coverage mapping: For each intended test scenario (e.g., fraud pattern, adverse drug event, edge-case loan application), verify that at least one synthetic record instantiates that scenario. Maintain a coverage matrix mapping test objectives to synthetic record counts. Gaps in coverage are defects, not unknowns.

• ▪Model behavioral equivalence (where source data is available): If reference models trained or evaluated on real data exist, compare key behavioral metrics (AUC, F1, calibration) when the same model is applied to real versus synthetic data. A meaningful degradation on synthetic data signals a fidelity problem that the statistical metrics may have missed. Define your degradation threshold before running the comparison.

• ▪Edge-case and adversarial scenario presence: Confirm that the synthetic dataset contains records designed to stress-test boundary conditions — maximum field values, rare but valid combinations, deliberately ambiguous inputs. These are not naturally abundant in statistically representative datasets; they may need to be engineered deliberately and documented as such.

• ▪Bias and fairness proxy evaluation: Run fairness metrics (demographic parity difference, equalized odds difference) on any model evaluated using the synthetic test set. Compare results to the same metrics run on real holdout data if available. A synthetic test set that produces artificially favorable fairness metrics will mask real-world disparate impact.

• ▪Usefulness to downstream consumers: Collect structured feedback from the data scientists, QE engineers, or model validators who used the synthetic data. Document specific gaps they encountered. This qualitative signal often catches failure modes that quantitative metrics miss.

Minimum pass threshold for Dimension 4: The test coverage matrix must be complete (no unmapped test objectives). A score of 0 on coverage mapping is an automatic fail.

---

Scoring Summary and Decision Protocol

Complete the following table for each synthetic dataset evaluated.

• ▪Dataset identifier and version
• ▪Generation method (e.g., GAN, VAE, rule-based, statistical sampling)
• ▪Source data description and date range
• ▪Intended use case(s)
• ▪Evaluator name and date

For each dimension, record: raw score (sum of criteria scores), maximum possible score, weighted score, and pass/fail against the minimum threshold.

Overall decision rules:

• ▪Any automatic fail criterion triggered → dataset does not proceed, regardless of total score.
• ▪Total weighted score below 60% of maximum → dataset does not proceed.
• ▪Total weighted score 60–79% → conditional approval with documented remediation plan and re-evaluation deadline.
• ▪Total weighted score 80% or above, no automatic fails → approved for intended use case; approval is use-case-specific and does not transfer to other applications without re-evaluation.

Note on weight adjustment: BFSI and insurance teams working on regulatory reporting models should consider increasing Dimension 2 (Business-Rule Conformance) weight to 35% and reducing Dimension 4 to 15%. Healthcare teams handling clinical AI should consider increasing Dimension 3 (Privacy Guarantees) weight to 35%. Document any weight adjustments and their rationale in the scorecard record.

---

Governance and Re-evaluation Triggers

A scorecard completed once is not a permanent approval. Synthetic data quality is conditional on the stability of the generation process, the source schema, and the intended use case. Establish a re-evaluation policy that triggers a new scorecard when any of the following occur:

• ▪The generative model is retrained, updated, or replaced.
• ▪The source data schema changes (new fields, changed distributions, altered business rules).
• ▪The intended use case expands or changes (e.g., a dataset approved for model testing is proposed for model training).
• ▪A downstream model or system using the synthetic data exhibits unexpected behavioral drift.
• ▪A regulatory update changes the applicable privacy or AI governance requirements.
• ▪More than a defined period (twelve months is a reasonable default estimate for most enterprise contexts) has elapsed since the last evaluation.

Store completed scorecards in your AI governance repository alongside model cards, risk assessments, and data lineage records. In an ISO/IEC 42001-aligned AI management system, synthetic data quality records belong in the operational controls documentation. Under the NIST AI RMF, they support the MEASURE and MANAGE functions.

---

A Note on Assurance as Practice

Synthetic data quality is not a one-time procurement check. It is a continuous assurance practice — one that requires the same rigor applied to model validation, code quality, and security review. The cost of a failed synthetic dataset is rarely visible at the point of generation; it materializes later, in a biased model, a missed regulatory defect, or a privacy incident that could have been anticipated.

The organizations that get this right treat synthetic data quality as a first-class engineering and governance concern: defined criteria, documented evidence, clear ownership, and scheduled review. This scorecard is a starting point for building that discipline into your AI assurance workflow.