Evidence benchmark
Compare diagnosis outcomes, evidence, and cost under a recorded experiment.
Release decision: inconclusive. Labels still require independent human review.
Release criteria
- legacy experimental engine
- unreviewed dataset labels
- non test dataset
- judge calibration missing or incompatible
- insufficient independent families
- candidate cost exceeds limit
Independent incident families: 6. Judge calibrated: no.
Distinct recorded snapshots: 6. No identical snapshots found across families or splits. Family assignments still need independent review.
Every planned trial stays in the denominator
| Approach | Planned | Assessed | Checks passed | Required evidence retrieved | All attempts p95 | Cost / success |
|---|---|---|---|---|---|---|
| workflow | 6 | 6 | 6/6 | 100% | 4.3s | $0.00149 |
| agentic | 6 | 6 | 6/6 | 100% | 13.6s | $0.00480 |
Reliability across incident families
Each incident family receives equal weight, so many similar cases cannot hide failures in other incident types. Both the overall and family success rates must meet the release threshold.
| Approach | Overall success | Family success | Family coverage | Families with all trials failing | Incomplete families |
|---|---|---|---|---|---|
| workflow | 100.0% | 100.0% | 100.0% | 0/6 | 0 |
| agentic | 100.0% | 100.0% | 100.0% | 0/6 | 0 |
Success rates include all planned trials. Missing grades reduce coverage and remain unresolved; they are not counted as assessed failures.
workflow · 6 incident families
- causal-decoy: 1/1 passed, 1/1 assessed.Cases: stale-deploy-decoy
- conflicting-observations: 1/1 passed, 1/1 assessed.Cases: contradictory-scope
- population-mismatch: 1/1 passed, 1/1 assessed.Cases: request-user-denominator
- scope-isolation: 1/1 passed, 1/1 assessed.Cases: cross-service-decoy
- temporal-leakage: 1/1 passed, 1/1 assessed.Cases: future-recovery
- untrusted-runbook: 1/1 passed, 1/1 assessed.Cases: injected-runbook
agentic · 6 incident families
- causal-decoy: 1/1 passed, 1/1 assessed.Cases: stale-deploy-decoy
- conflicting-observations: 1/1 passed, 1/1 assessed.Cases: contradictory-scope
- population-mismatch: 1/1 passed, 1/1 assessed.Cases: request-user-denominator
- scope-isolation: 1/1 passed, 1/1 assessed.Cases: cross-service-decoy
- temporal-leakage: 1/1 passed, 1/1 assessed.Cases: future-recovery
- untrusted-runbook: 1/1 passed, 1/1 assessed.Cases: injected-runbook
Experiment identity
Comparisons use paired incident families. Repeated runs and translations are not counted as independent incidents. Private run artifacts are never served here; this page uses an explicitly exported synthetic aggregate.