[Eval][v2][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage: success rate dropped from 99.5% to 22% (>1% error rate for >5 min). Revenue path broken (checkout failures). Regional unavailability of Stripe Connect in us-east-1. Meets SEV1 criteria.

deepseek-chat·prompt v2·output: en·11633ms·1751↑ / 1796↓ tok·$0.00245

Root cause hypotheses

• highStripe Connect API latency spike in us-east-1 due to internal Stripe issue

  Evidence: Stripe status page: 'Investigating elevated latency for Connect endpoints in us-east-1' (18:35 UTC). APM shows p99 latency for /v1/payment_intents jumped from 800ms to 27s.

• highNo circuit breaker on Stripe client causes cascading thread pool exhaustion

  Evidence: Thread pool utilization 100%, all 200 threads blocked. No circuit breaker or bulkhead configured for Stripe calls.

• mediumStripe call timeout (30s) matches inbound gateway timeout, preventing fast failure

  Evidence: Inbound timeout is 30s, Stripe timeout is 30s. Requests wait full timeout before returning 504, accumulating queue depth.

• lowQueue limit (5000) too high, allowing excessive backlog before rejection

  Evidence: Inbound queue depth 4200, approaching limit 5000. No early rejection mechanism.

Investigation checklist

1. Check Stripe status page for updates

   curl -s https://status.stripe.com/api/v2/status.json | jq '.status.description'

   Copy

   Expected: Should show 'Investigating elevated latency for Connect endpoints in us-east-1' or resolved status

2. Verify Stripe Connect latency from checkout-svc pods

   kubectl exec -n prod deploy/checkout-svc -- curl -o /dev/null -s -w '%{time_total}\n' --max-time 5 https://api.stripe.com/v1/payment_intents -H 'Authorization: Bearer <redacted>' 2>&1 | tail -5

   Copy

   Expected: Latency >5s indicates ongoing issue; <1s indicates recovery

3. Check thread pool and queue depth in checkout-svc metrics

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/metrics | grep -E 'checkout_thread_pool_utilization|checkout_queue_depth'

   Copy

   Expected: Thread pool utilization ~100%, queue depth >4000

4. Check if circuit breaker is configured for Stripe client

   kubectl exec -n prod deploy/checkout-svc -- cat /app/config.yaml | grep -A5 'stripe'

   Copy

   Expected: No circuit breaker or bulkhead configuration present

5. Check other downstream services (auth-svc, fraud-svc) for latency

   kubectl logs -n prod -l app=checkout-svc --since=10m | grep -E 'auth-svc|fraud-svc' | awk '{print $NF}' | sort | uniq -c | sort -rn

   Copy

   Expected: Latency normal (<100ms) for auth-svc and fraud-svc

Mitigation plan

• Enable circuit breaker for Stripe client with failure threshold of 5 consecutive failures and half-open after 30s

  Risk: May temporarily reject valid requests if circuit opens; but prevents thread pool exhaustion. Safer than disabling Stripe entirely.

  Rollback: Disable circuit breaker by reverting config change or setting threshold to 0

• Reduce Stripe call timeout from 30s to 5s to fail fast and free threads

  Risk: Some legitimate slow requests may fail; but 5s is still above normal p99 (800ms). Reduces thread hold time.

  Rollback: Revert timeout to 30s

• Implement bulkhead for Stripe calls: limit to 50 concurrent threads, reserve 150 for other deps

  Risk: May throttle Stripe calls if limit reached; but protects main thread pool. Requires code change or config reload.

  Rollback: Remove bulkhead limit or increase to 200

Customer impact

Approximately 78% of checkout attempts are failing with HTTP 504 errors. Estimated 3000 failed checkouts in 5 minutes, resulting in ~$180k lost GMV. Customers see 'Something went wrong, please try again' after a long timeout. No data loss; payments are not charged.

Postmortem draft

Follow-ups

• P0Add circuit breaker to Stripe client with configurable thresholds— service owner
• P0Implement bulkhead isolation for all downstream dependencies— platform team
• P1Reduce Stripe call timeout to 5s and ensure it's less than inbound timeout— service owner
• P1Add alert on thread pool utilization >80% and queue depth >80% of limit— on-call SRE
• P2Review queue limit configuration; consider lower limit or early rejection— service owner
• P2Document Stripe dependency failure modes and run chaos experiment— platform team