[Eval][v2][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage: checkout success rate dropped from 99.5% to 22% (>1% error rate for >5 min). Revenue path broken: estimated $180k GMV lost. Regional unavailability: Stripe us-east-1 outage affecting all users.

deepseek-chat·prompt v2·output: en·12984ms·1751↑ / 2078↓ tok·$0.00276

Root cause hypotheses

• highStripe Connect API latency spike due to upstream outage in us-east-1

  Evidence: Stripe status page: 'Investigating elevated latency for Connect endpoints in us-east-1' posted 18:35 UTC. APM shows p99 latency for /v1/payment_intents jumped from 800ms to 27s at 18:40 UTC.

• highNo circuit breaker on Stripe client causes cascading thread pool exhaustion

  Evidence: Thread pool utilization at 100% (200 threads blocked). Inbound queue depth 4200. No circuit breaker or bulkhead configured for Stripe calls.

• mediumStripe timeout (30s) matches inbound gateway timeout, preventing fast failure

  Evidence: p99 latency 28s, hitting 30s timeout. No shorter timeout for Stripe calls to fail fast.

• lowRecent code change increased Stripe call volume or changed retry behavior

  Evidence: No direct evidence in context. Could be investigated via recent deployments or config changes.

• lowNetwork issue between our cluster and Stripe (e.g., DNS, TLS, routing)

  Evidence: No evidence of network errors; Stripe status page confirms their issue. Low likelihood given external confirmation.

Investigation checklist

1. Check Stripe status page for updates

   curl -s https://status.stripe.com/api/v2/status.json | jq '.status.description'

   Copy

   Expected: Should show 'Investigating elevated latency for Connect endpoints in us-east-1' or updated status

2. Check thread pool and queue depth in checkout-svc

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/actuator/health | jq '.details.threadPool'

   Copy

   Expected: threadPool.activeCount == 200, queueDepth > 4000

3. Check APM for Stripe call latency breakdown

   curl -s 'http://apm-dashboard.internal/d/checkout?var-service=checkout-svc&var-downstream=stripe-connect&from=now-30m' | jq '.series[0].values[-1][1]'

   Copy

   Expected: p99 latency > 25s

4. Verify other downstream services are healthy

   kubectl logs -n prod -l app=auth-svc --since=15m | grep -c 'error'

   Copy

   Expected: error count < 10 (normal baseline)

5. Check recent deployments to checkout-svc

   kubectl rollout history -n prod deploy/checkout-svc

   Copy

   Expected: Last change > 1 hour ago (no recent deploy)

6. Check Stripe API key and connectivity

   kubectl exec -n prod deploy/checkout-svc -- curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 https://api.stripe.com/v1/payment_intents

   Copy

   Expected: 200 or 401 (not timeout)

Mitigation plan

• Enable circuit breaker for Stripe calls with fast-fail timeout (e.g., 5s) and fallback to error response

  Risk: May cause partial checkout failures if circuit opens, but prevents thread pool exhaustion. Safer than disabling Stripe entirely.

  Rollback: Disable circuit breaker via feature flag: kubectl set env deploy/checkout-svc CIRCUIT_BREAKER_ENABLED=false

• Reduce Stripe call timeout from 30s to 5s to fail fast and free threads

  Risk: May cause more timeouts if Stripe latency is transient, but threads will be released quickly. Low risk.

  Rollback: Revert timeout: kubectl set env deploy/checkout-svc STRIPE_TIMEOUT=30

• Implement bulkhead for Stripe calls (max 20 concurrent threads) to protect main thread pool

  Risk: May limit throughput to Stripe, but prevents total exhaustion. Requires code change or config reload.

  Rollback: Remove bulkhead: kubectl set env deploy/checkout-svc STRIPE_BULKHEAD_MAX=200

• If Stripe outage persists >15 min, disable Stripe payment path and fall back to alternative payment provider or show maintenance page

  Risk: Lost revenue from Stripe payments; customers may be redirected. Destructive if not coordinated with business.

  Rollback: Re-enable Stripe: kubectl set env deploy/checkout-svc STRIPE_ENABLED=true

Customer impact

Approximately 78% of checkout attempts are failing with a 504 error. In the last 5 minutes, about 3,000 checkouts have failed, resulting in an estimated $180,000 in lost gross merchandise value. Users see a timeout error page after 30 seconds. No ETA for resolution; Stripe is investigating their own issue.

Postmortem draft

Follow-ups

• P0Add circuit breaker to Stripe client with 5s timeout and fallback— service owner
• P0Implement bulkhead isolation for Stripe calls (max 20 concurrent threads)— service owner
• P1Reduce Stripe call timeout to 5s (configurable via env var)— service owner
• P1Create runbook for Stripe outage scenario with step-by-step mitigation— on-call SRE
• P1Add alert on thread pool utilization > 80%— platform team
• P2Review and test fallback payment provider integration— service owner
• P1Postmortem meeting within 48 hours— on-call SRE