Adversarial Safety Evaluation Harness and Release-Gating Governance

You are the evaluation platform lead designing the adversarial safety harness for models deployed via [INSERT MODEL PLATFORM], governed under authorization [INSERT AUTHORIZATION ID]. In-scope models: [INSERT MODEL LIST]; the harness supports release gating, continuous regression, and vendor comparison, using only licensed/synthetic evaluation content. Module 1 - Evaluation taxonomy: hazard categories aligned to current public safety taxonomies, capabilities/behaviors under test, language and modality coverage, and explicit exclusions. Module 2 - Test-library governance: versioned, owned test sets, anti-contamination controls (holdout sets, rotation), change review, and no-training-on-eval enforcement. Module 3 - Attack-strategy coverage: abstract, policy-derived strategies (refusal testing, multi-turn, encoding, cross-lingual, agent tool abuse) parameterized against templates, with safe-content guardrails. Module 4 - Orchestration: reproducible runs (seeds, temperature, endpoints), rate limits, parallelism, cost controls, and signed result artifacts stored in [INSERT RESULTS STORE]. Module 5 - Judgment design: multi-model judge ensemble calibrated against human labels, confidence reporting, bias audits, and sampled human adjudication; document judge prompt versions. Module 6 - Metrics and gating: pass rates by category with confidence intervals, regression thresholds versus [INSERT BASELINE], aggregate safety score, waiver workflow, and mandatory human review for high-risk capabilities. Module 7 - Protective follow-through: automatic tickets to model owners, a guardrail-tuning backlog, and trend dashboards for [INSERT RISK GOVERNANCE BODY]. Include a reference architecture text diagram, a CI integration example, and a run-manifest schema. Define controls preventing eval misuse: access control, redaction, safe harbor for test content, and audit logging of who ran what. Specify vendor/model acceptance comparison criteria for procurement, requiring equivalent evidence from third-party models. Map metrics to [INSERT FRAMEWORKS, e.g., NIST AI RMF MEASURE, EU AI Act GPAI, ISO/IEC 42001, MLCommons safety guidance]. Document limitations: benchmark overfitting, judge drift, distribution shift; require periodic benchmark refresh and external assessment; define harness SLOs, reproducibility tolerances, and quarterly taxonomy updates. Present the taxonomy and gating policy first for approval, then harness design, runbooks, and dashboard specs in phases. ================