Agent and MCP Tool Evaluation Harness with CI Gates

You are an evaluation engineer. Build an evaluation harness for agents and MCP tools, covering offline trajectories, regression gates, and leaderboards for [INSERT PRODUCT]. Suites range from unit tool tests to end-to-end coding tasks scored by programmatic checks combined with judge models, at a cadence of [INSERT WEEKLY RUN VOLUME]. Deliver in modules, waiting for approval each time. Module 1 Architecture: runner, dataset store, scorer pipeline, and result warehouse, with a mermaid end-to-end diagram. Module 2 Data model: suites, cases (including secret-held golden cases), variants (model/prompt/tool versions), runs and scores. Module 3 Case authoring: YAML case format, MCP-server fixtures, deterministic fake clocks and network, and a case-lint CLI. Module 4 Execution: parallel sharded runs, retries reported separately from first-try metrics, model-provider abstraction, and cost caps. Module 5 Tool metrics: schema validity, latency percentiles, error classes, and idempotency violations captured through an MCP proxy. Module 6 Agent metrics: task success, steps-to-success, trajectory precision, token cost, and calibrated judge-model rubrics. Module 7 Analysis: paired comparisons with confidence intervals, scorecards, variant diff views, and automatic regression bisection. Module 8 CI gate: a pipeline plugin that blocks releases when [INSERT PROMOTION METRIC] regresses beyond a configured threshold. Module 9 Human review: sampling UI, inter-rater-agreement tracking, and adjudication workflows for disputed judge scores. Module 10 Red-team cases: prompt-injection and tool-confusion suites continuously seeded from production incidents. Module 11 Meta-testing: run known-bad agents against the harness to prove it detects seeded defects before it is trusted. Module 12 Deployment: Kubernetes runner pools, encrypted datasets, retention policy for [INSERT RETENTION WINDOW], and dashboards. Provide case-authoring guidelines so new teams can add reliable suites without inflating scores. Ask five questions first, including which single score is the promotion gate for agents. ================