Summary

Severity reasoning: User-facing outage with 8% error rate and 15s latency, revenue impact on Black Friday, and potential data inconsistency from DB overload. Full outage of a critical customer-facing service.

deepseek-chat·prompt v1·output: en·12413ms

Root cause hypotheses

• highCache stampede from simultaneous key expiration at 09:00 UTC

  Evidence: Redis cache miss rate for 'catalog:item:*' jumped to 95% at 09:00 UTC, coinciding with TTL expiry of pre-warmed keys (set at 02:00 UTC, TTL=7h).

• highDB connection pool exhaustion due to surge of uncached queries

  Evidence: Logs show DB connection pool hit ceiling at 09:00:02 UTC, queries started queueing. Postgres CPU at 100% sustained.

• highBlack Friday traffic spike overwhelming DB capacity

  Evidence: Traffic ramp of 12x normal at 09:00 UTC from marketing email blast, combined with cache miss, caused DB overload.

• mediumMissing per-key locking or rate limiting on cache miss

  Evidence: No per-key locking; every cache miss triggers a fresh DB query, leading to stampede. This is a design flaw.

Investigation checklist

1. Confirm cache miss rate and key expiration pattern

   redis-cli --raw -h <redis-host> INFO keyspace | grep catalog && redis-cli --raw -h <redis-host> TTL catalog:item:12345

   Copy

   Expected: High miss rate and TTL near 0 for sample keys.

2. Check DB connection pool usage and active queries

   kubectl exec -n prod catalog-svc-0 -- psql -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"

   Copy

   Expected: Active connections near max_pool size (e.g., 100/100).

3. Identify slow queries contributing to CPU saturation

   kubectl exec -n prod catalog-svc-0 -- psql -c "SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;"

   Copy

   Expected: Top queries are SELECT from catalog_item table with high total_time.

4. Check if DB is CPU-bound or I/O-bound

   kubectl exec -n prod catalog-svc-0 -- psql -c "SELECT pg_stat_get_backend_pid(s.backendid), s.wait_event_type FROM pg_stat_activity s WHERE s.state = 'active';"

   Copy

   Expected: Most wait_event_type are 'CPU' or 'LWLock' (lock contention).

5. Verify Redis is not the bottleneck

   redis-cli --raw -h <redis-host> INFO CPU | grep used_cpu_sys

   Copy

   Expected: CPU usage low (25% as reported).

Mitigation plan

• Immediately shed load by rate-limiting catalog-svc at the ingress gateway to 50% of current traffic

  Risk: Some users will see 429 errors, but prevents total outage. Destructive if not reverted.

  Rollback: Gradually increase rate limit back to 100% after cache is warmed.

• Manually warm Redis cache with a script that reads from a DB replica (not primary) to avoid further load

  Risk: If replica is also overloaded, this could worsen. Use a dedicated read-only replica if available.

  Rollback: Kill the warm-up script if DB load increases beyond 90%.

• Extend TTL on existing keys to prevent immediate re-expiry (won't help in-flight but prevents second wave)

  Risk: Stale data may be served for longer, but acceptable during incident.

  Rollback: Set TTL back to original after incident.

Customer impact

Customers on Black Friday are experiencing 15-second load times and intermittent 503 errors when viewing product detail pages. Approximately 8% of requests fail. No data loss, but degraded experience for all users hitting catalog-svc.

Postmortem draft

Follow-ups

• P0Implement per-key locking (e.g., Redis SETNX) to prevent stampede— catalog-svc owner
• P0Stagger cache TTLs with jitter to avoid simultaneous expiry— catalog-svc owner
• P1Add rate limiting at ingress gateway for catalog-svc— platform team
• P1Provision a read-only DB replica for cache warm-up scripts— DB team
• P2Review and update incident response runbook for cache stampede— on-call SRE