[Eval][v3][en] Regional 5xx spike after DNS TTL change

Summary

Severity reasoning: User-facing outage: 35% error rate (exceeds 1% threshold) for >5 minutes, affecting a major region (us-west-2). Revenue path likely broken due to high error rate. This meets SEV1 criteria.

deepseek-chat·prompt v3·output: en·10540ms·2035↑ / 1585↓ tok·$0.00229

Root cause hypotheses

• highNew NLB not deployed in us-west-2, causing DNS resolution failures for pods that flush cache

  Evidence: Logs show nginx in us-west-2 getting NXDOMAIN or stale IPs for internal-services.example.com. us-east-1 works because NLB exists there.

• highDNS TTL reduction from 300s to 30s caused rapid cache expiration, exposing the missing NLB in us-west-2

  Evidence: Change occurred at 21:40 UTC; incident started around 22:10 UTC, consistent with TTL expiration window. 30% of pods still working (cached old ALB IP).

• mediumOld ALB DNS records were removed, preventing fallback resolution

  Evidence: Context states 'its DNS records were not removed' but later says 'NXDOMAIN' which implies removal. Contradiction; likely records were removed or not published.

• lowDNS propagation delay causing inconsistent resolution across pods

  Evidence: Some pods have cached old IPs and work; others fail. This is consistent with gradual cache expiration, not propagation delay.

Investigation checklist

1. Verify DNS resolution for internal-services.example.com from us-west-2 pods

   kubectl exec -n prod -l app=api-gateway -- nslookup internal-services.example.com 2>&1 | head -20

   Copy

   Expected: Should return NXDOMAIN or an IP not in us-west-2 (e.g., us-east-1 NLB IP). If old ALB IP appears, cache is stale.

2. Check if new NLB exists in us-west-2

   aws elbv2 describe-load-balancers --region us-west-2 --names new-nlb-name 2>&1

   Copy

   Expected: Should return 'LoadBalancerNotFound' if not deployed.

3. Check nginx error logs for upstream connection failures

   kubectl logs -n prod -l app=api-gateway --since=30m | grep -i 'upstream timed out' | tail -20

   Copy

   Expected: Multiple entries showing connection timeouts to upstream.

4. Check DNS TTL configuration for internal-services.example.com

   dig internal-services.example.com +short +ttlid 2>&1

   Copy

   Expected: TTL should be 30s (new value) or 300s (old).

5. Verify old ALB is still operational

   aws elbv2 describe-target-health --region us-west-2 --target-group-arn <old-alb-tg-arn> 2>&1

   Copy

   Expected: Targets should be healthy if ALB is still up.

Mitigation plan

• Revert DNS change: point internal-services.example.com back to old ALB CNAME with original TTL (300s)

  Risk: Low risk; old ALB is still up. May cause brief propagation delay.

  Rollback: Re-apply the new DNS change after fixing NLB deployment.

• Deploy new NLB in us-west-2 and update DNS to point to it

  Risk: Medium risk; takes time to create NLB and update DNS. During creation, outage continues.

  Rollback: Revert DNS to old ALB if new NLB fails.

• Hot-patch us-west-2 pods to use us-east-1 NLB IP directly via /etc/hosts or environment variable

  Risk: Medium risk; cross-region latency increase, potential for asymmetric routing. Requires pod restart.

  Rollback: Remove the patch and revert to DNS resolution.

Customer impact

Approximately 35% of requests to us-west-2 are failing with HTTP 502 errors. Users experience timeouts or errors when accessing services. us-east-1 users are unaffected. No ETA yet.

Postmortem draft

Follow-ups

• P0Ensure new NLB is deployed in all regions before DNS cutover— infrastructure team
• P1Add pre-change validation checklist for DNS migrations— platform team
• P1Improve monitoring to detect regional DNS resolution failures— observability team
• P2Review DNS change process to require multi-region verification— SRE team