[Eval][v2][en] Order service OOM crashloop following v3.7 deploy

Summary

Severity reasoning: User-facing outage: checkout success rate >1% (3.2% errors) for >5 min, revenue path broken (checkout failures), and error rate exceeds 1% threshold. Meets SEV1 criteria.

deepseek-chat·prompt v2·output: en·10713ms·1756↑ / 1582↓ tok·$0.00221

Root cause hypotheses

• highIn-process request cache (Map-based, no eviction) introduced in v3.7 grows unboundedly, consuming all pod memory.

  Evidence: Heap profile shows ~340Mi held by Map keyed on request ID, never cleared. Memory trend shows linear growth from 180Mi to 500Mi over 5 hours, then OOM.

• mediumMemory limit (512Mi) is too low for the new cache, but cache is intended to be bounded.

  Evidence: Pre-deploy baseline was stable at 200Mi. Cache was added without eviction policy, so it's not a limit issue but a leak.

• lowExternal dependency (e.g., database connection pool) is leaking connections, causing memory pressure.

  Evidence: No connection pool errors in logs; heap profile shows Map as primary consumer, not connection objects.

Investigation checklist

1. Check current pod status and memory usage across all replicas.

   kubectl top pod -n prod -l app=order-svc --sort-by=memory

   Copy

   Expected: Memory usage near 500Mi for running pods; OOMKilled pods show 0 memory.

2. Verify the heap profile from a captured pod to confirm Map memory dominance.

   kubectl exec -n prod -l app=order-svc -- sh -c 'curl -s localhost:6060/debug/pprof/heap?debug=1 | head -100'

   Copy

   Expected: Output shows 'Map' entries consuming ~340Mi, with no eviction logic.

3. Check deploy changelog to confirm the cache addition and absence of eviction.

   kubectl rollout history deployment/order-svc -n prod --revision=3 | grep -i cache

   Copy

   Expected: Line: 'added in-process request cache (Map-based, no eviction)'

4. Monitor memory growth rate on a running pod to confirm linear trend.

   kubectl exec -n prod -l app=order-svc -- sh -c 'while true; do cat /sys/fs/cgroup/memory/memory.usage_in_bytes; sleep 60; done'

   Copy

   Expected: Memory increases by ~1-2Mi per minute, consistent with leak.

5. Check if any recent traffic spike correlates with faster growth.

   kubectl logs -n prod -l app=order-svc --since=1h | grep -c 'request'

   Copy

   Expected: Request count per minute; compare with memory growth rate.

Mitigation plan

• Rollback order-svc to previous version (v3.6) to remove the leaky cache.

  Risk: Rollback will cause brief downtime during pod replacement; cache benefits lost. Safe operation.

  Rollback: Re-deploy v3.7 if rollback causes issues; or re-apply with fixed cache.

• If rollback not immediately possible, increase memory limit to 1Gi to buy time.

  Risk: Higher memory usage may cause node pressure; temporary fix only.

  Rollback: Revert memory limit to 512Mi after rollback or fix.

• Scale up HPA max replicas to 16 to spread load and reduce per-pod memory pressure.

  Risk: More pods may increase database connection count; monitor for connection pool exhaustion.

  Rollback: Reduce max replicas back to 12.

Customer impact

Approximately 25% of checkout requests are failing or timing out. Users see 'Checkout failed' errors. Estimated 3% of all users affected. No data loss. ETA for full recovery: 30 minutes after rollback.

Postmortem draft

Follow-ups

• P0Add eviction policy (TTL or LRU) to in-process cache before re-enabling.— service owner
• P1Set up memory usage alert at 80% of limit to warn before OOM.— on-call SRE
• P1Add memory leak detection test in CI (e.g., heap profile diff).— platform team
• P2Review deploy process for performance regression testing.— service owner