[Eval][v3][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage: error rate > 1% (78% failure rate) for >5 minutes, revenue path broken (checkout failures causing $180k GMV loss). Matches SEV1 criteria.

deepseek-chat·prompt v3·output: en·12428ms·2002↑ / 1869↓ tok·$0.00260

Root cause hypotheses

• highStripe Connect API latency spike causing thread pool exhaustion in checkout-svc

  Evidence: APM shows Stripe /v1/payment_intents p99 jumped from 800ms to 27s; Stripe status page confirms elevated latency for Connect endpoints in us-east-1; thread pool utilization at 100% with all 200 workers blocked.

• highNo circuit breaker on Stripe client leads to cascading failures

  Evidence: Configuration review shows no circuit breaker or bulkhead for Stripe calls; all threads block waiting for Stripe response, causing queue buildup.

• lowInbound request surge exacerbates thread pool exhaustion

  Evidence: Inbound queue depth at 4200 (limit 5000), but success rate drop correlates with Stripe latency increase; no evidence of abnormal traffic spike before incident.

• mediumStripe API timeout misconfigured (30s) too high, causing long-held threads

  Evidence: Timeout matches inbound gateway timeout; reducing it could free threads faster, but not root cause of Stripe latency.

Investigation checklist

1. Confirm Stripe latency via APM and status page

   curl -s https://status.stripe.com/api/v1/incidents | jq '.incidents[] | select(.status != "resolved")'

   Copy

   Expected: Active incident for Connect endpoints in us-east-1 with elevated latency

2. Check thread pool utilization and queue depth in checkout-svc

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/actuator/threaddump | grep -E 'pool-.*thread.*state=BLOCKED' | wc -l

   Copy

   Expected: 200 blocked threads (all workers)

3. Verify downstream dependencies (auth-svc, fraud-svc) are healthy

   kubectl logs -n prod -l app=auth-svc --since=10m | grep -iE 'error|timeout' | tail -5

   Copy

   Expected: No errors or timeouts

4. Check Stripe API response times from checkout-svc logs

   kubectl logs -n prod -l app=checkout-svc --since=15m | grep -oP 'stripe_request_duration_ms=\K\d+' | sort -n | tail -5

   Copy

   Expected: Values near 27000 (27s) indicating high latency

5. Review circuit breaker and bulkhead configuration for Stripe client

   kubectl exec -n prod deploy/checkout-svc -- cat /app/config/application.yml | grep -A10 'stripe'

   Copy

   Expected: No circuit breaker or bulkhead settings; only timeout=30s

Mitigation plan

• Enable circuit breaker for Stripe client with fallback to error page or retry later

  Risk: May cause immediate failures for all Stripe-dependent checkouts if circuit opens; but prevents thread exhaustion.

  Rollback: Disable circuit breaker via feature flag or config reload.

• Reduce Stripe call timeout from 30s to 5s to free threads faster

  Risk: May cause more timeouts if Stripe latency is high, but reduces thread hold time.

  Rollback: Revert timeout to 30s via config update.

• Implement bulkhead isolation for Stripe calls (dedicated thread pool of 20 threads)

  Risk: Requires code change and redeploy; not immediate. Could be done as hotfix if deployment pipeline allows.

  Rollback: Revert to shared thread pool via config.

• If Stripe outage persists, disable Stripe payment path and show 'payment unavailable' message

  Risk: Lost revenue from Stripe payments, but preserves checkout flow for other payment methods.

  Rollback: Re-enable Stripe path when Stripe recovers.

Customer impact

Approximately 78% of checkout attempts are failing with a 504 error. Customers see a timeout or error page when trying to complete purchases. Estimated 3000 failed checkouts in the last 5 minutes, with $180k in lost GMV. No ETA for resolution yet.

Postmortem draft

Follow-ups

• P0Implement circuit breaker for Stripe client with fallback— payments-platform
• P0Add bulkhead isolation for Stripe calls (dedicated thread pool)— payments-platform
• P1Reduce Stripe call timeout from 30s to 5s— on-call SRE
• P1Add monitoring alerts for thread pool utilization >80% and queue depth >80%— platform team
• P2Review all external dependency configurations for circuit breakers and bulkheads— service owner
• P2Update runbook for Stripe dependency failure with mitigation steps— on-call SRE