[Eval][v2][en] Cache stampede after Redis key expiry on Black Friday morning

Summary

Severity reasoning: User-facing outage: p99 latency 15s (baseline 80ms), 503 error rate 8% > 1% for >5 min, revenue path broken (product detail pages essential for Black Friday sales).

deepseek-chat·prompt v2·output: en·12977ms

Root cause hypotheses

• highCache TTL of 7h caused all keys to expire at 09:00 UTC, triggering a stampede under 12x traffic.

  Evidence: Redis cache miss rate for 'catalog:item:*' jumped from 2% to 95% at 09:00 UTC; last cache hit at 08:59:58 UTC.

• highDB connection pool exhausted by concurrent queries from cache misses, causing queueing and latency.

  Evidence: DB CPU 100% sustained, lock waits climbing, connection pool hit ceiling at 09:00:02 UTC.

• mediumMissing per-key locking or rate limiting on cache miss DB queries allowed all requests to hit DB simultaneously.

  Evidence: No per-key locking in code; every cache miss triggers a fresh DB query without coordination.

• mediumPre-warm cache at 02:00 UTC with 7h TTL was insufficient for Black Friday traffic ramp starting at 09:00.

  Evidence: Marketing email blast at 09:00 UTC caused 12x traffic ramp; cache was not refreshed to cover the peak.

• lowDatabase query performance degraded due to missing indexes or inefficient queries under load.

  Evidence: No direct evidence; DB CPU is 100% but could be from simple queries under high concurrency.

Investigation checklist

1. Check Redis cache miss rate for catalog keys to confirm stampede.

   redis-cli -h redis-cache -p 6379 INFO stats | grep -i 'keyspace_misses'

   Copy

   Expected: Keyspace miss rate > 90% (currently 95%)

2. Check Postgres active queries and wait events.

   psql -h pg-primary -U postgres -c "SELECT pid, query_start, state, wait_event_type, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start LIMIT 20;"

   Copy

   Expected: Many queries with state 'active' and wait_event_type 'Lock' or 'IO'

3. Check DB connection pool usage.

   kubectl exec -n prod deployment/catalog-svc -- curl -s localhost:8080/actuator/metrics/hikaricp.connections.active

   Copy

   Expected: Active connections near max pool size (e.g., 50/50)

4. Check catalog-svc error logs for 503s.

   kubectl logs -n prod -l app=catalog-svc --since=15m | grep -i '503' | head -20

   Copy

   Expected: Lines with HTTP 503 status codes

5. Verify cache TTL configuration.

   kubectl exec -n prod deployment/catalog-svc -- cat /app/config/application.yml | grep -A2 'cache'

   Copy

   Expected: TTL: 7h for catalog:item keys

Mitigation plan

• Shed load by temporarily scaling down non-critical traffic: block catalog-svc requests from internal batch jobs via network policy.

  Risk: Batch jobs may fail or retry, but no user impact. Blast radius limited to internal systems.

  Rollback: Remove network policy to restore traffic.

• Increase DB connection pool size temporarily to reduce queueing (e.g., from 50 to 100).

  Risk: May increase DB CPU further; monitor closely. Blast radius: DB performance.

  Rollback: Revert pool size to original value.

• Manually warm cache by running a script that re-queries DB for popular items and sets cache keys with staggered TTLs.

  Risk: Script will add load to DB; run with rate limiting (e.g., 10 queries/sec). Blast radius: DB load.

  Rollback: Kill the script if DB CPU exceeds 95%.

• Enable per-key locking (e.g., Redis SETNX) to prevent concurrent DB queries for the same key.

  Risk: Code change requires deployment; may introduce latency if lock contention high. Blast radius: catalog-svc.

  Rollback: Revert the code change and redeploy.

Customer impact

Users on Black Friday are experiencing slow product page loads (up to 15 seconds) and intermittent errors (503) when viewing items. Approximately 12x normal traffic is affected. No ETA yet.

Postmortem draft

Follow-ups

• P0Implement staggered cache TTLs to prevent simultaneous expiration.— service owner
• P0Add per-key locking (e.g., Redis SETNX) for cache miss DB queries.— service owner
• P1Add rate limiting for cache miss DB queries to prevent stampede.— service owner
• P1Review and adjust cache pre-warm schedule for peak traffic days.— platform team
• P1Add alert on cache miss rate exceeding threshold (e.g., >10%).— on-call SRE
• P2Document incident response steps for cache stampede scenarios.— on-call SRE