Summary

Severity reasoning: User-facing outage with error rate >1% (8% 503s) for >5 min, revenue path broken (product detail pages failing), and regional unavailability risk. Matches SEV1 rubric: error rate >1% for >5 min and revenue path broken.

deepseek-chat·prompt v3·output: en·14749ms·2056↑ / 2291↓ tok·$0.00308

Root cause hypotheses

• highCache stampede due to simultaneous key expiration at 09:00 UTC

  Evidence: Redis cache miss rate for 'catalog:item:*' keys jumped from 2% to 95% at 09:00 UTC. Pre-warm TTL=7h from 02:00 UTC expired exactly at 09:00 UTC. Logs show last cache hit at 08:59:58, first miss at 09:00:00.

• highDB connection pool exhaustion from concurrent cache miss queries

  Evidence: Postgres CPU 100% sustained, lock waits climbing, connection pool hit ceiling at 09:00:02 UTC. Queries started queueing. 8% 503s indicate pool exhaustion.

• highBlack Friday traffic surge amplified cache miss impact

  Evidence: Traffic ramp 12x normal at 09:00 UTC due to marketing email blast. Normal cache hit ratio would have absorbed this, but with 95% miss rate, all traffic hit DB.

• mediumMissing per-key locking or request coalescing for cache misses

  Evidence: No per-key locking; every cache miss triggers a fresh DB query. This is a design gap that allowed stampede to overwhelm DB.

• lowDB query performance degradation due to lock contention

  Evidence: Lock waits climbing indicates queries are blocking each other. Could be due to heavy read load or inefficient queries, but primary cause is volume.

Investigation checklist

1. Check Redis cache miss rate for catalog keys

   redis-cli -h redis-cache -p 6379 INFO stats | grep -i 'keyspace_misses' && redis-cli -h redis-cache -p 6379 --raw GET 'catalog:item:*' | head -5

   Copy

   Expected: Miss rate near 100% for catalog keys; keys missing from cache.

2. Check Postgres active queries and wait events

   psql -h pg-primary -U app -c "SELECT pid, query_start, state, wait_event_type, wait_event, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start LIMIT 20;"

   Copy

   Expected: Many queries waiting on 'LWLock' or 'IO' with long query_start times.

3. Check DB connection pool utilization

   psql -h pg-primary -U app -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';" && psql -h pg-primary -U app -c "SHOW max_connections;"

   Copy

   Expected: Active connections near max_connections (e.g., 100/100).

4. Check catalog-svc error logs for 503s

   kubectl logs -n prod -l app=catalog-svc --since=15m | grep -i '503' | tail -20

   Copy

   Expected: Lines indicating 'connection refused' or 'timeout' from DB.

5. Verify cache TTL configuration

   kubectl exec -n prod deploy/catalog-svc -- cat /app/config.yaml | grep -A2 'cache.*ttl'

   Copy

   Expected: TTL set to 7h for catalog keys.

6. Check Redis CPU and memory

   redis-cli -h redis-cache -p 6379 INFO cpu | grep 'used_cpu_sys' && redis-cli -h redis-cache -p 6379 INFO memory | grep 'used_memory_human'

   Copy

   Expected: CPU low (~25%), memory usage normal.

Mitigation plan

• Shed load by temporarily blocking non-critical traffic (e.g., marketing redirects) at the load balancer or via feature flag.

  Risk: May affect marketing campaigns but preserves core catalog functionality.

  Rollback: Remove the block rule or disable feature flag.

• Manually warm the cache by running a script to re-populate catalog keys from DB, but with rate limiting to avoid further DB overload.

  Risk: If rate limit is too high, DB may become more saturated. Safer to run with low concurrency.

  Rollback: Stop the script; keys will expire naturally.

• Increase DB connection pool size temporarily (e.g., from 100 to 200) to reduce queuing.

  Risk: May increase DB CPU further; monitor closely. Could cause OOM if memory is tight.

  Rollback: Revert to original pool size.

• Extend cache TTL for existing keys (e.g., set TTL to 24h) to prevent immediate re-expiry.

  Risk: Does not help in-flight requests; only prevents future stampedes. May cause stale data.

  Rollback: Reset TTL to original value.

• Enable request coalescing for cache misses (e.g., using Redis SETNX or a mutex) to deduplicate DB queries.

  Risk: Requires code change; not immediate. Could introduce latency if not implemented correctly.

  Rollback: Revert code change.

Customer impact

Users are experiencing slow loading and intermittent errors (503) when viewing product detail pages. Approximately 8% of requests are failing, and page load times have increased from under 100ms to over 15 seconds. This affects all users during the Black Friday sales event.

Postmortem draft

Follow-ups

• P0Add jitter to cache TTLs to prevent simultaneous key expiration— catalog-svc team
• P0Implement request coalescing for cache misses using Redis SETNX or similar— catalog-svc team
• P1Add load shedding at load balancer for cache stampede scenarios— platform team
• P1Review and improve pre-warm strategy (staggered refresh, shorter TTL with background refresh)— catalog-svc team
• P1Add monitoring and alert for cache miss rate spikes— observability team
• P2Document cache stampede runbook and mitigation steps— on-call SRE