[Eval][v3][en] Cache stampede after Redis key expiry on Black Friday morning

Summary

Severity reasoning: User-facing outage: p99 latency 15s (baseline 80ms) and 503 error rate 8% > 1% for >5 minutes, affecting all product detail page requests. Revenue path is broken as catalog-svc is critical for Black Friday shopping. This meets SEV1 criteria.

deepseek-chat·prompt v3·output: en·15943ms·2056↑ / 2154↓ tok·$0.00292

Root cause hypotheses

• highCache stampede due to simultaneous key expiration at 09:00 UTC under high traffic

  Evidence: Redis cache miss rate for 'catalog:item:*' jumped to 95% at 09:00 UTC (baseline 2%). Last cache hit at 08:59:58 UTC. TTL=7h from 02:00 UTC pre-warm aligns with expiration at 09:00 UTC.

• highPostgres connection pool exhaustion from concurrent cache miss queries

  Evidence: Logs at 09:00:02 UTC show 'DB connection pool hits ceiling, queries start queueing'. Postgres CPU at 100% sustained with lock waits climbing.

• lowDatabase query performance degradation due to missing index or slow query

  Evidence: No direct evidence of missing indexes; baseline p99 was 80ms. The sudden spike at 09:00 UTC correlates with cache expiration, not a schema change. Unlikely.

• lowNetwork latency or resource contention on database host

  Evidence: No network errors reported. CPU is 100% due to query load, not I/O wait. Likely not a network issue.

• lowRedis instance failure or misconfiguration causing cache misses

  Evidence: Redis CPU is 25% (healthy), and cache miss rate is specific to catalog keys. Redis itself is operational. Unlikely.

Investigation checklist

1. Check current Redis cache miss rate for catalog keys

   redis-cli -h redis-catalog -p 6379 INFO stats | grep -E 'keyspace_misses|keyspace_hits'

   Copy

   Expected: keyspace_misses should be high (e.g., >90%) if stampede ongoing; if low, stampede may have subsided.

2. Check Postgres active queries and lock waits

   psql -h pg-catalog -U app -c "SELECT pid, query_start, state, wait_event_type, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start LIMIT 20;"

   Copy

   Expected: Many queries with state 'active' and wait_event_type 'LWLock' or 'Lock' indicating contention.

3. Check Postgres connection pool usage

   psql -h pg-catalog -U app -c "SELECT count(*) FROM pg_stat_activity WHERE state != 'idle';"

   Copy

   Expected: Count near max_connections (e.g., 200) indicating pool exhaustion.

4. Check catalog-svc error logs for 503s

   kubectl logs -n prod -l app=catalog-svc --since=15m | grep -E 'HTTP 503|timeout|connection refused' | tail -50

   Copy

   Expected: Multiple 503 lines with timestamps after 09:00 UTC.

5. Verify cache TTL configuration

   kubectl exec -n prod deploy/catalog-svc -- cat /app/config.yaml | grep -A2 'cache.*ttl'

   Copy

   Expected: TTL set to 7h (25200s) for catalog items.

6. Check if any recent deployment changed cache behavior

   kubectl rollout history -n prod deploy/catalog-svc

   Copy

   Expected: No recent changes; last rollout >24h ago.

Mitigation plan

• Shed load by rate-limiting catalog-svc at ingress (e.g., nginx) to reduce DB pressure

  Risk: Some users will see 429 errors, but prevents complete outage. Blast radius: all catalog requests.

  Rollback: Remove rate limit by reverting ingress config change.

• Manually warm Redis cache by running a script to re-populate popular catalog items with a longer TTL (e.g., 12h)

  Risk: Script may increase DB load further if not throttled. Blast radius: DB CPU may spike temporarily.

  Rollback: Stop the script; cache will expire naturally.

• Increase Postgres connection pool size temporarily (e.g., from 100 to 200) to handle queued queries

  Risk: May increase CPU/memory pressure on DB. Blast radius: DB host resources.

  Rollback: Revert connection pool size to original value.

• Extend cache TTL for catalog items to 12h to prevent near-future expiration during peak

  Risk: Stale data may be served for up to 12h if items change. Blast radius: catalog freshness.

  Rollback: Revert TTL to 7h.

Customer impact

Users are experiencing slow loading (up to 15 seconds) and intermittent errors when viewing product detail pages. Approximately 12x normal traffic from a Black Friday marketing campaign is affected. Estimated 100% of users hitting catalog pages are impacted. No ETA yet.

Postmortem draft

Follow-ups

• P0Implement staggered cache TTL with random jitter to prevent simultaneous expiration— catalog-svc team
• P0Add per-key locking (e.g., Redis SETNX) for cache misses to reduce DB load— catalog-svc team
• P1Set up circuit breaker for catalog-svc DB queries to shed load automatically— platform team
• P1Review pre-warm schedule and TTL for Black Friday and other peak events— catalog-svc team
• P1Add alert on cache miss rate spike to detect stampedes earlier— on-call SRE
• P2Document incident response steps for cache stampede in runbook— on-call SRE