[Eval][v2][en] Cache stampede after Redis key expiry on Black Friday morning

Summary

Severity reasoning: User-facing outage: p99 latency 15s (baseline 80ms), 503 error rate 8% > 1% for >5 min, revenue path broken (product detail pages are critical for Black Friday sales). This meets SEV1 criteria.

deepseek-chat·prompt v2·output: en·15341ms·1805↑ / 1870↓ tok·$0.00254

Root cause hypotheses

• highCache stampede due to simultaneous key expiration and traffic surge

  Evidence: Redis cache miss rate for 'catalog:item:*' jumped from 2% to 95% at 09:00 UTC, coinciding with TTL expiry of pre-warmed keys (set at 02:00 UTC, TTL=7h). DB CPU hit 100% immediately after.

• highDB connection pool exhaustion from concurrent queries

  Evidence: Logs at 09:00:02 UTC show connection pool hitting ceiling and queries queueing. Lock waits climbing.

• mediumMissing per-key locking or rate limiting on cache miss

  Evidence: No evidence of locking in code; every cache miss triggers a fresh DB query. This is a design gap.

• lowDatabase query performance degradation (e.g., missing index, slow query)

  Evidence: No specific slow query identified yet; DB CPU is 100% but could be due to volume rather than individual slow queries.

• lowNetwork latency or DNS issue causing retries

  Evidence: No network errors in logs; Redis CPU is normal (25%), ruling out network as primary cause.

Investigation checklist

1. Check Redis cache miss rate for catalog keys

   redis-cli -h redis-cache -p 6379 INFO stats | grep -i 'keyspace_misses'

   Copy

   Expected: High miss rate (>90%) confirms stampede

2. Check DB active connections and wait events

   SELECT count(*), state, wait_event_type FROM pg_stat_activity WHERE state != 'idle' GROUP BY state, wait_event_type;

   Copy

   Expected: Many connections in 'active' state with wait_event_type 'Lock' or 'IO'

3. Check DB CPU usage

   kubectl top pod -n prod -l app=catalog-db --containers | grep -i postgres

   Copy

   Expected: CPU usage near 100%

4. Check catalog-svc error logs for 503s

   kubectl logs -n prod -l app=catalog-svc --since=10m | grep -E 'HTTP 503|timeout|connection refused' | tail -20

   Copy

   Expected: Multiple 503 errors due to upstream DB timeout

5. Verify cache key TTL and expiry pattern

   redis-cli -h redis-cache -p 6379 TTL catalog:item:popular | head -5

   Copy

   Expected: TTL near 0 or negative for many keys

6. Check if any recent deployment changed cache logic

   kubectl rollout history deployment/catalog-svc -n prod

   Copy

   Expected: No recent changes; stampede is operational

Mitigation plan

• Shed load by temporarily blocking non-critical traffic to catalog-svc (e.g., disable product recommendations via feature flag)

  Risk: Reduces feature functionality but protects DB; may impact user experience but prevents total outage

  Rollback: Re-enable feature flag once DB CPU drops below 70%

• Increase Redis cache TTL for catalog keys to 12h to prevent immediate re-expiry

  Risk: Stale data for up to 12h; acceptable during Black Friday

  Rollback: Revert TTL to 7h after incident

• Manually warm cache by running a script to re-populate popular keys from DB (if DB load allows)

  Risk: Running additional DB queries could worsen load; only do if DB CPU < 80%

  Rollback: Kill the warm-up script if DB CPU exceeds 90%

• Scale up catalog-db read replicas to handle query load

  Risk: May take minutes to provision; temporary cost increase

  Rollback: Scale down after cache warms

Customer impact

Users browsing product detail pages on the website and mobile app are experiencing slow load times (up to 15 seconds) and intermittent errors (503). This affects all users during Black Friday shopping, estimated 12x normal traffic. No data loss has occurred.

Postmortem draft

Follow-ups

• P0Implement per-key locking (e.g., SETNX) on cache miss to prevent stampede— catalog-svc team
• P0Stagger cache TTLs with jitter to avoid simultaneous expiry— catalog-svc team
• P1Add circuit breaker for DB queries from catalog-svc— platform team
• P1Improve pre-warm job to align with traffic patterns and add random TTL offset— data engineering team
• P1Set up alert on cache miss rate spike (e.g., >50% deviation from baseline)— observability team
• P2Review and update incident response runbook for cache stampede scenarios— on-call SRE