Rubric v1 · 5 dims · LLM-as-judge (DeepSeek). Each row is an analysis scored 1–5 per dim.
Total spend
$0.2770
n = 118
Avg per analysis
$0.00235
Avg latency
11560ms
↑ / ↓ tokens
231.6k / 195.0k
Excludes the streaming /api/analyze path — usage isn't reachable from that flow (limitation, not a bug). Includes batch evals + scenario runs + re-runs.
| key | n | overall | Specificity | Safety | Actionability | Domain correctness | Completeness |
|---|---|---|---|---|---|---|---|
| v1 | 49 | 4.58 | 4.61 | 4.04 | 4.61 | 4.92 | 4.71 |
| v2 | 52 | 4.41 | 4.54 | 3.98 | 4.54 | 4.81 | 4.17 |
| v3 | 34 | 4.58 | 4.71 | 4.00 | 4.71 | 4.79 | 4.68 |
| key | n | overall | Specificity | Safety | Actionability | Domain correctness | Completeness |
|---|---|---|---|---|---|---|---|
| en | 67 | 4.60 | 4.72 | 4.07 | 4.72 | 4.91 | 4.60 |
| zh | 68 | 4.42 | 4.50 | 3.94 | 4.50 | 4.78 | 4.40 |
Solid incident response with good specificity and domain correctness, but safety could be improved by adding safer-first alternatives, and completeness is hindered by placeholders in the postmortem.
Strong incident response with concrete evidence and actionable steps; minor gaps in command specificity and postmortem completeness.
A thorough and well-structured incident response with strong specificity, safety, actionability, domain correctness, and completeness. Minor improvements could include more explicit time ranges in commands and safer-first alternatives for mitigations.
Excellent incident response with high specificity, actionability, and domain correctness; minor safety and completeness gaps prevent a perfect score.
Strong incident response with high specificity, actionability, and domain correctness. Minor safety concern and placeholder gaps in postmortem prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor gaps in postmortem placeholders and a slight safety concern with the manual warm script.
A thorough and well-structured incident response with strong domain correctness and completeness, though minor improvements in specificity and safety would elevate it further.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor improvement: consider safer-first alternative before increasing DB pool size.
A thorough and well-structured incident response with concrete evidence and actionable steps, scoring high across all dimensions.
A thorough and well-structured incident response with strong specificity and domain correctness. Minor improvements in safety (safer DB scaling) and actionability (script details) would make it excellent.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety concern with manual warm-up script risk, but overall outstanding.
A thorough and well-structured incident response with strong specificity, safety, and domain correctness. Minor improvements could include more detailed commands and a pre-written warm script.
A strong, well-structured incident response with clear root cause and actionable steps. Minor gaps in specificity (relative time window) and completeness (missing timeline details) prevent a perfect score.
A strong incident response with high specificity, safety, and domain correctness. Minor improvements in actionability and completeness would make it excellent.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
A thorough and well-structured incident response that scores highly across all dimensions, with minor room for improvement in specificity and actionability of some commands.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement possible but overall outstanding.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement possible for hot-patch step.
Strong incident response with high specificity and actionability; minor safety and completeness gaps.
Excellent response with high specificity, actionability, and domain correctness; minor safety and completeness gaps prevent a perfect score.
Strong response with high specificity and actionability; minor safety and completeness gaps prevent a perfect score.
Strong incident response with clear root cause and actionable steps; minor gaps in specificity and completeness due to placeholders.
Strong incident response with clear root cause identification and actionable steps. Minor improvements in specificity and completeness would make it excellent.
A strong, well-structured incident response with high specificity and domain correctness. Minor improvements could include filling placeholders and adding time ranges to commands.
A strong incident response with specific commands and clear mitigation. Minor improvements: replace placeholders for full copy-pasteability and adjust severity to SEV1.
A strong, well-structured incident response with clear root cause analysis and actionable steps. Minor improvements in command specificity and actionability would make it excellent.
A strong, well-structured incident response with clear evidence and actionable steps. Minor improvements in command specificity and severity classification would make it excellent.
A strong, well-structured incident response with clear root cause analysis and actionable steps. Minor improvements in specificity and completeness would make it excellent.
A thorough and well-structured incident response that is specific, safe, actionable, correct, and complete. Minor improvements could include fully copy-pasteable commands and explicit safer-first alternatives.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
A strong, well-structured incident response with clear evidence and actionable steps. Minor improvements could include more precise time ranges in commands and fewer placeholders in the postmortem.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement possible by adding safer-first alternatives.
A strong, well-structured incident response with concrete evidence and actionable steps, slightly held back by minor placeholders and a few less-immediate actions.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety concern with destructive fallback but overall outstanding.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor improvement: consider adding a shorter timeout (e.g., 8s) as a mitigation.
Strong incident response with clear evidence and actionable steps; minor gaps in specificity and completeness prevent top scores.
Strong response with high specificity and domain correctness; safety and actionability are good but could be improved with more precise time ranges and fully filled postmortem sections.
Excellent incident response with high specificity, actionability, and domain correctness. Minor gaps in completeness (postmortem placeholders) and safety (acceptable risk in mitigation).
A strong incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor improvements: add time ranges to all commands, fill in postmortem placeholders, and provide more detailed risk descriptions.
Excellent response with high specificity, safety, actionability, and domain correctness; completeness slightly reduced due to placeholder sections in postmortem draft.
A strong, well-structured incident response with high specificity, safety, and domain correctness. Minor improvements could include fully fleshing out the postmortem and adding more detail on enabling the circuit breaker.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
A strong, well-structured incident response with clear evidence and actionable steps. Minor improvements could include more precise commands and fuller postmortem sections.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety concern with circuit breaker activation but rollback provided.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Strong incident response with high specificity and actionability; minor safety concern and placeholder in postmortem draft.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor placeholder in postmortem timeline prevents a perfect 5 in completeness.
A strong, well-structured incident response with clear evidence and actionable steps. Minor improvements: add time ranges to commands, ensure all rollback commands are explicit, and calibrate severity to SEV2.
Excellent response with high specificity and actionability; minor severity misclassification prevents a perfect score.
Excellent response with high specificity and actionability. Minor severity misclassification prevents perfect domain correctness score.
Excellent incident response with high specificity, actionability, and completeness. Minor issue in severity classification and a slight distraction with fast-json-stringify, but overall very strong.
Excellent response with high specificity, actionability, and completeness. Minor safety concern and severity misclassification prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety concern on scaling down HPA but overall outstanding.
Strong incident response with high specificity and actionability; minor issues in severity classification and postmortem completeness.
Excellent response with high specificity, actionability, and completeness. Minor issue in severity classification and one hypothesis not fully eliminated.
Strong incident response with concrete evidence and actionable steps; minor issues in specificity (placeholder pod names) and domain correctness (severity misclassification).
Strong response with high specificity and actionability; minor severity misclassification and placeholder sections prevent a perfect score.
Strong incident response with high specificity and actionability; minor safety and completeness gaps prevent a perfect score.
Strong incident response with high specificity and actionability; minor gaps in safety (no explicit drain) and completeness (postmortem placeholders).
Strong incident response with clear root cause identification and actionable steps. Minor improvements in command specificity and timeline completeness would make it excellent.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
A strong, well-structured incident response with clear evidence and actionable steps. Minor improvements could include fully copy-pasteable commands and a more detailed rollback plan for the memory limit increase.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Strong incident response with high specificity and actionability; minor safety concern and placeholder sections in postmortem prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Exemplary incident response with high specificity, safety, actionability, domain correctness, and completeness.
A strong, well-structured incident response with high specificity and actionability; minor safety and completeness gaps prevent a perfect score.
Strong incident response with high specificity and actionability; minor safety and completeness gaps prevent a perfect score.
Excellent incident response with high specificity, actionability, and domain correctness. Minor gaps in safety (missing safer-first alternative for kill query) and completeness (postmortem placeholders).
Strong incident response with high specificity and actionability; minor safety concern and incomplete postmortem draft prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent response with high specificity, safety, actionability, and domain correctness; minor completeness gaps in postmortem draft place it at 4 rather than 5.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, actionability, and domain correctness. Minor safety concern with query killing but mitigated by read-only nature.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Strong incident response with high specificity and domain correctness; minor safety and completeness gaps prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement could be adding a safer alternative before increasing DB pool size.
Strong incident response with clear evidence and actionable steps; minor gaps in specificity and safety prevent a perfect score.
Strong incident response with high specificity, safety, and domain correctness; minor gaps in actionability (script details) and completeness (placeholders in postmortem).
Excellent incident response with high specificity, correctness, and completeness. Minor safety improvement could be adding a safer-first alternative before manual cache warmup.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety concern with connection pool increase but rollback provided.
Strong incident response with high domain correctness and specificity; minor gaps in actionability and completeness due to placeholder fields and missing exact parameters.
A strong, well-structured incident response with clear evidence and actionable steps. Minor improvements could include more precise time ranges in commands and filling postmortem placeholders.
A strong incident response with clear root cause and actionable steps; minor gaps in command completeness and postmortem fill-ins.
A strong incident response with high specificity and domain correctness; minor gaps in actionability and completeness due to placeholder values and thin sections.
A solid incident response with good specificity and actionability, but the severity misclassification and a slightly risky mitigation action prevent a higher score.
A strong, well-structured incident response with clear root cause analysis and actionable steps. Minor improvements could include filling placeholders and providing hosted zone IDs for full specificity.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor improvement: consider safer-first alternative before disabling Stripe.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement could be adding safer-first alternatives before destructive changes.
Strong incident response with high specificity and domain correctness; minor gaps in safety (lack of safer-first alternatives) and completeness (placeholder sections).
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety quibble but overall outstanding.
A strong incident response with high specificity and domain correctness; minor gaps in safety (lack of safer-first alternative) and completeness (postmortem placeholders) prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement possible by adding safer-first alternatives before disabling payment path.
Excellent response with high specificity, actionability, and completeness; minor safety and domain correctness issues.
Excellent response with high specificity, actionability, and completeness. Minor severity misclassification and safety could be improved with safer-first alternatives.
Strong response with concrete evidence and actionable steps; minor severity misclassification and lack of safer-first alternatives prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor severity misclassification does not detract from overall quality.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Strong incident response with high specificity and actionability; minor gaps in safety (no safer-first alternative for index) and completeness (placeholders).
Excellent incident response with high specificity, safety, actionability, and domain correctness. Minor completeness gaps with placeholders in timeline and thin customer impact.
Exemplary incident response with high specificity, safety, actionability, domain correctness, and completeness.
A strong incident response with clear root cause identification and actionable steps; minor gaps in specificity of time ranges and incomplete postmortem sections prevent a perfect score.
Excellent response with high specificity, actionability, and domain correctness. Minor safety and completeness gaps prevent a perfect score.
Strong incident response with clear root cause, actionable steps, and good safety considerations. Minor improvements could include adding time ranges to commands and expanding customer impact.
Excellent incident response with strong specificity, safety, actionability, domain correctness, and completeness. Minor improvements in command completeness and safer-first alternatives would make it perfect.
Excellent response with high specificity, safety, actionability, and domain correctness; minor gaps in completeness (postmortem placeholders) and safety (no safer-first alternative for code change).
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement possible by adding a safer-first alternative for cache warmup.
A thorough, well-structured incident response with strong evidence and actionable steps. Minor improvements could include fully copy-pasteable commands and a provided warm-up script.
Strong response with precise root cause and actionable steps; minor gaps in command specificity and timeline completeness.
A strong incident response with high specificity and domain correctness; minor gaps in safety (missing pre-revert health check) and completeness (placeholders in postmortem) prevent a perfect score.
A strong incident response with clear root cause analysis and actionable steps. Minor improvements in specificity (time windows) and completeness (filling placeholders) would make it excellent.
A thorough and well-structured incident response with strong domain correctness and completeness, slightly held back by minor specificity gaps in commands.
Strong incident response with clear root cause analysis and actionable steps. Minor gaps in command specificity and postmortem completeness prevent a perfect score.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness. Minor safety improvement possible by adding safer-first alternatives for thread pool increase.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Strong response with high specificity and actionability; minor issues in severity classification and postmortem completeness.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Excellent response with high specificity, actionability, and domain correctness. Minor safety concern and placeholder gaps prevent a perfect score.
A strong, detailed incident response with excellent specificity and actionability; minor gaps in safety (no memory check before increasing max_connections) and completeness (placeholders in postmortem).
Strong incident response with high specificity and actionability; minor safety concern with database restart and incomplete postmortem placeholders.
Excellent incident response with high specificity, safety, actionability, domain correctness, and completeness.
Strong incident response with clear evidence and actionable steps. Minor improvements in specificity and completeness would make it excellent.
A strong incident response with concrete evidence and actionable steps, but minor gaps in safety (no safer alternative before terminating connections) and completeness (postmortem placeholders) prevent a perfect score.
A strong, well-structured incident response with clear evidence and actionable steps, though safety could be improved by prioritizing less destructive mitigations.