[Eval][v1][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage with revenue impact (estimated $180k GMV lost). Full degradation of checkout for 78% of users, meeting SEV1 criteria.

deepseek-chat·prompt v1·output: en·10404ms·1370↑ / 1638↓ tok·$0.00217

Root cause hypotheses

• highStripe Connect API latency spike in us-east-1 causes thread pool exhaustion

  Evidence: APM shows Stripe /v1/payment_intents p99 jumped from 800ms to 27s; Stripe status page confirms elevated latency for Connect endpoints in us-east-1; thread pool utilization at 100%

• highNo circuit breaker on Stripe client allows unbounded retries or queuing

  Evidence: Config review shows no circuit breaker; thread pool exhaustion consistent with all threads blocking on Stripe calls

• mediumStripe call timeout (30s) matches inbound timeout, causing cascading failures

  Evidence: p99 latency 28s hits 30s timeout; inbound queue depth 4200/5000 indicates requests piling up

• highShared thread pool without bulkhead allows Stripe latency to block all checkout requests

  Evidence: No bulkhead; Stripe calls share main worker pool; all 200 threads blocked

Investigation checklist

1. Confirm Stripe latency via APM

   curl -s 'http://apm-dashboard:8080/api/v1/services/checkout-svc/transactions?filter=stripe' | jq '.latency.p99'

   Copy

   Expected: Value > 20s, confirming Stripe is the bottleneck

2. Check thread pool metrics

   kubectl exec deploy/checkout-svc -- curl -s localhost:8080/metrics | grep -E 'thread_pool_active|thread_pool_queue'

   Copy

   Expected: active threads = 200, queue depth > 4000

3. Verify Stripe status page

   curl -s 'https://status.stripe.com/api/v1/incidents?filter=us-east-1' | jq '.incidents[0].status'

   Copy

   Expected: "investigating" or "identified" for Connect endpoints

4. Check if circuit breaker exists in Stripe client config

   kubectl exec deploy/checkout-svc -- cat /app/config.yaml | grep -A5 'stripe'

   Copy

   Expected: No circuit_breaker or bulkhead settings present

5. Check inbound queue depth trend

   kubectl exec deploy/checkout-svc -- curl -s localhost:8080/metrics | grep 'inbound_queue_depth'

   Copy

   Expected: Value near 5000 (queue limit), indicating backpressure

Mitigation plan

• Enable circuit breaker on Stripe client with failure threshold 5, timeout 10s, half-open after 30s

  Risk: May temporarily reject valid payments if Stripe recovers slowly; no data loss

  Rollback: Revert config change and restart checkout-svc pods

• Reduce Stripe call timeout from 30s to 5s to fail fast and free threads

  Risk: Some legitimate payments may fail if Stripe latency >5s; no data loss

  Rollback: Revert timeout to 30s and restart pods

• If Stripe outage persists, disable Stripe payment path and fall back to 'payment failed' page

  Risk: All Stripe payments will fail; no data loss; users can retry later

  Rollback: Re-enable Stripe path via feature flag

Customer impact

Approximately 78% of checkout attempts are failing with HTTP 504 errors. Users cannot complete purchases using any payment method that goes through Stripe. Estimated 3000 failed checkouts in 5 minutes, with $180k GMV lost. No ETA for resolution; Stripe is investigating.

Postmortem draft

Follow-ups

• P0Add circuit breaker to Stripe client with proper thresholds— service owner
• P1Implement bulkhead isolation for Stripe calls— service owner
• P1Reduce Stripe call timeout to 5s— service owner
• P1Add alert on thread pool utilization >80%— on-call SRE
• P2Review all downstream dependencies for circuit breaker and bulkhead patterns— platform team
• P2Update runbook for Stripe outage scenarios— on-call SRE