[Eval][v2][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage: checkout success rate < 1% (22%) for >5 min, revenue path broken (Stripe payment processing), and regional unavailability of Stripe Connect impacting all users. Meets SEV1 criteria.

deepseek-chat·prompt v2·output: en·10517ms·1751↑ / 1669↓ tok·$0.00231

Root cause hypotheses

• highStripe Connect API latency spike due to upstream issue in us-east-1

  Evidence: Stripe status page: 'Investigating elevated latency for Connect endpoints in us-east-1' (18:35 UTC). APM shows p99 latency for /v1/payment_intents jumped from 800ms to 27s.

• highNo circuit breaker on Stripe client causing cascading thread pool exhaustion

  Evidence: Thread pool utilization at 100% (200 workers blocked), inbound queue depth 4200. Stripe calls share main worker pool with no bulkhead.

• mediumStripe timeout (30s) matches inbound gateway timeout, preventing fast failure

  Evidence: p99 latency 28s, hitting 30s timeout. No shorter timeout or circuit breaker to fail fast.

• lowRecent deployment or config change to checkout-svc increased Stripe call volume

  Evidence: No direct evidence in context; low likelihood as Stripe status page indicates upstream issue.

Investigation checklist

1. Check Stripe status page for updates

   curl -s https://status.stripe.com/ | grep -i 'connect'

   Copy

   Expected: Status should show 'Investigating' or 'Resolved' for Connect endpoints.

2. Verify Stripe latency from checkout-svc pods

   kubectl exec -n prod deploy/checkout-svc -- curl -o /dev/null -s -w '%{time_total}\n' https://api.stripe.com/v1/payment_intents -H 'Authorization: Bearer <redacted>'

   Copy

   Expected: Response time should be <1s normally; if >10s, confirms Stripe latency.

3. Check thread pool and queue depth in checkout-svc

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/actuator/threadpool | jq '.threadPool.utilization, .queueDepth'

   Copy

   Expected: Utilization should be <80%; queue depth <1000. Currently 100% and 4200.

4. Check for recent deployments to checkout-svc

   kubectl rollout history deployment -n prod checkout-svc

   Copy

   Expected: No recent changes; if recent, check diff.

5. Check downstream dependencies (auth-svc, fraud-svc) latency

   kubectl exec -n prod deploy/checkout-svc -- curl -o /dev/null -s -w '%{time_total}\n' http://auth-svc:8080/health

   Copy

   Expected: Response time <100ms; if elevated, indicates broader issue.

Mitigation plan

• Enable circuit breaker on Stripe client with fallback to fail fast (return 503 immediately) to protect thread pool

  Risk: All Stripe payments will fail fast, but thread pool will recover. No data loss.

  Rollback: Disable circuit breaker via feature flag or config reload.

• Reduce Stripe timeout from 30s to 5s to fail fast and free threads

  Risk: Some legitimate payments may timeout if Stripe latency is high but not all requests are slow. Safer than current state.

  Rollback: Revert timeout to 30s via config push.

• Implement bulkhead for Stripe calls (dedicated thread pool of 20 threads) to isolate from main pool

  Risk: Requires code change; cannot be done quickly. Not recommended for immediate mitigation.

  Rollback: Revert to shared pool via config.

Customer impact

Approximately 78% of checkout attempts are failing with HTTP 504 errors. ~3000 failed checkouts in 5 minutes, estimated $180k lost GMV. Customers cannot complete purchases. No ETA for resolution; dependent on Stripe recovery.

Postmortem draft

Follow-ups

• P0Add circuit breaker to Stripe HTTP client with fallback— checkout-svc service owner
• P0Implement bulkhead (dedicated thread pool) for Stripe calls— checkout-svc service owner
• P1Reduce Stripe API timeout from 30s to 5s— checkout-svc service owner
• P1Add alert on thread pool utilization >80%— platform team
• P2Review and improve incident response runbook for Stripe dependencies— on-call SRE
• P2Evaluate adding Stripe status page as a monitoring source— platform team