[Eval][v2][en] Cache stampede after Redis key expiry on Black Friday morning

Summary

Severity reasoning: User-facing outage: p99 latency 15s (baseline 80ms), 503 error rate 8% > 1% for >5 min, revenue path broken (Black Friday traffic, checkout affected).

deepseek-chat·prompt v2·output: en·10347ms·1805↑ / 1680↓ tok·$0.00234

Root cause hypotheses

• highCache TTL of 7h caused all keys to expire at 09:00 UTC, triggering a stampede under 12x traffic.

  Evidence: Redis cache miss rate for 'catalog:item:*' jumped from 2% to 95% at 09:00 UTC; last cache hit at 08:59:58 UTC.

• highDB connection pool exhaustion due to concurrent queries from cache misses.

  Evidence: DB CPU 100%, lock waits climbing, connection pool ceiling hit at 09:00:02 UTC.

• mediumMissing per-key locking or rate limiting on cache miss regeneration.

  Evidence: No per-key locking; every cache miss triggers a fresh DB query, confirmed by logs.

• lowBlack Friday traffic spike alone overwhelmed DB without cache stampede.

  Evidence: Traffic is 12x normal, but Redis CPU is only 25%, indicating cache miss is the primary cause.

• lowDatabase query performance degradation (e.g., missing index, slow query).

  Evidence: No specific slow query identified; DB CPU is 100% but lock waits suggest contention, not slow queries.

Investigation checklist

1. Check Redis cache miss rate for catalog keys.

   redis-cli -h redis-cache -p 6379 INFO stats | grep -i 'keyspace_misses'

   Copy

   Expected: High miss rate (>90%) confirms stampede.

2. Check DB active queries and locks.

   SELECT pid, query_start, state, wait_event_type, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start LIMIT 20;

   Copy

   Expected: Many queries waiting on locks or running concurrently.

3. Check catalog-svc error logs for 503s.

   kubectl logs -n prod -l app=catalog-svc --since=15m | grep -i '503' | head -20

   Copy

   Expected: Intermittent 503 errors due to DB timeouts.

4. Verify cache TTL configuration.

   kubectl exec -n prod deploy/catalog-svc -- cat /app/config.yaml | grep -A2 'cache.*ttl'

   Copy

   Expected: TTL set to 7h (25200s).

5. Check DB connection pool metrics.

   kubectl exec -n prod deploy/catalog-svc -- curl localhost:8080/metrics | grep 'db_connections'

   Copy

   Expected: Connection pool exhausted (max connections reached).

Mitigation plan

• Shed load by temporarily blocking non-critical traffic (e.g., product detail page for non-featured items) via feature flag or rate limiter.

  Risk: May degrade user experience for some users; ensure checkout flow is unaffected.

  Rollback: Disable the feature flag or increase rate limit back to normal.

• Manually warm the cache by running a script that re-queries DB for popular items and sets cache keys with a jittered TTL.

  Risk: Additional DB load during warm-up; must be done gradually to avoid further overload.

  Rollback: Stop the warm-up script; cache will repopulate naturally over time.

• Increase DB connection pool size temporarily (if DB has capacity) to reduce queuing.

  Risk: May increase DB CPU further; monitor closely.

  Rollback: Revert connection pool size to original value.

Customer impact

Users on Black Friday are experiencing slow product detail page loads (up to 15 seconds) and intermittent errors (503). Checkout may be affected. Estimated impact: all users browsing product pages, ~100% of traffic during peak.

Postmortem draft

Follow-ups

• P0Implement jittered TTL for cache keys to prevent simultaneous expiration.— service owner
• P0Add per-key locking (e.g., Redis SETNX) for cache miss regeneration.— service owner
• P1Add rate limiting for cache miss queries to DB.— service owner
• P1Set up pre-warming for high-traffic events (e.g., Black Friday).— platform team
• P2Improve alerting to include cache miss rate threshold.— on-call SRE
• P2Review DB connection pool sizing and add auto-scaling.— platform team