[Eval][v2][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage: success rate dropped from 99.5% to 22% (>1% error rate for >5 min). Revenue path broken: estimated $180k GMV lost. Regional unavailability: checkout service effectively down for most users.

deepseek-chat·prompt v2·output: en·14767ms

Root cause hypotheses

• highStripe Connect API latency spike causing thread pool exhaustion

  Evidence: APM shows Stripe /v1/payment_intents p99 jumped from 800ms to 27s; thread pool utilization 100%; Stripe status page confirms elevated latency in us-east-1 since 18:35 UTC.

• highNo circuit breaker on Stripe client causing cascading failures

  Evidence: Config review shows no circuit breaker or bulkhead for Stripe calls; all threads block waiting on Stripe, leading to queue buildup (4200 requests).

• mediumStripe API timeout mismatch causing retry storms

  Evidence: Inbound gateway timeout is 30s, Stripe call timeout is also 30s; no client-side retry policy visible; but if retries exist, they could amplify load.

• lowResource exhaustion on checkout-svc pods (CPU/memory)

  Evidence: Thread pool exhaustion is the primary symptom, but CPU/memory metrics not provided; could be secondary factor.

• lowNetwork issue between checkout-svc and Stripe

  Evidence: Stripe status page indicates their own issue; network path unlikely given other services normal.

Investigation checklist

1. Check Stripe status page for updates

   curl -s https://status.stripe.com/api/v2/status.json | jq '.status.description'

   Copy

   Expected: Should show 'Investigating elevated latency for Connect endpoints in us-east-1' or resolved status.

2. Verify thread pool exhaustion in checkout-svc logs

   kubectl logs -n prod -l app=checkout-svc --since=20m | grep -i 'thread pool' | tail -20

   Copy

   Expected: Lines indicating all threads busy, queue full, or rejected tasks.

3. Check APM for Stripe dependency latency breakdown

   curl -s 'http://apm-dashboard:8080/api/v1/services/checkout-svc/dependencies/stripe?time=18:30-18:50' | jq '.p99_latency_ms'

   Copy

   Expected: p99 latency around 27000ms (27s) vs baseline 800ms.

4. Inspect Stripe client configuration for circuit breaker and timeout

   kubectl exec -n prod deploy/checkout-svc -- cat /app/config/stripe.yaml | grep -E 'timeout|circuit|bulkhead|retry'

   Copy

   Expected: No circuit breaker or bulkhead settings; timeout likely 30s.

5. Check inbound queue depth and request rate

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/metrics | grep -E 'queue_depth|request_rate'

   Copy

   Expected: queue_depth near 4200, request_rate stable or increasing.

6. Verify other downstream services are healthy

   kubectl logs -n prod -l app=auth-svc --since=10m | grep -c 'error' && kubectl logs -n prod -l app=fraud-svc --since=10m | grep -c 'error'

   Copy

   Expected: Low error counts (0-5) indicating auth and fraud are normal.

7. Check if Stripe API key or rate limit issues exist

   kubectl logs -n prod -l app=checkout-svc --since=20m | grep -iE 'rate limit|429|too many requests' | head -10

   Copy

   Expected: No rate limit errors; Stripe issue is latency, not throttling.

Mitigation plan

• Enable circuit breaker on Stripe client to fail fast and shed load

  Risk: May cause partial checkout failures for users if circuit opens; but better than total outage. Safer than disabling Stripe entirely.

  Rollback: Disable circuit breaker by setting threshold to 0 or removing config; restart pods.

• Increase thread pool size temporarily to handle queued requests

  Risk: May increase resource pressure (CPU/memory) on pods; could cause OOM if too large. Monitor closely.

  Rollback: Revert thread pool size to original value and restart pods.

• Implement bulkhead isolation for Stripe calls to prevent thread pool exhaustion

  Risk: Requires code change and deployment; not immediate. For now, use circuit breaker as faster mitigation.

  Rollback: Remove bulkhead config and restart.

• Reduce Stripe call timeout to 5s to fail fast and avoid queue buildup

  Risk: May cause false positives if Stripe latency is transient; but 5s is reasonable for payment intents. Safer than 30s.

  Rollback: Revert timeout to 30s and restart pods.

Customer impact

Approximately 78% of checkout attempts are failing with HTTP 504 errors. Users see a timeout or error page after waiting up to 30 seconds. Estimated 3000 failed checkouts in 5 minutes, resulting in $180k lost GMV. No data loss; payments are not processed. ETA depends on Stripe recovery.

Postmortem draft

Follow-ups

• P0Implement circuit breaker on Stripe client with configurable threshold and half-open recovery— service owner
• P0Add bulkhead isolation for Stripe calls to separate thread pool— service owner
• P1Reduce Stripe call timeout from 30s to 5s— service owner
• P1Add alert on thread pool utilization >80% and queue depth >1000— on-call SRE
• P2Evaluate fallback payment provider for critical path— platform team
• P1Update incident runbook for Stripe dependency failures with circuit breaker steps— on-call SRE
• P2Review Stripe API retry policy and ensure exponential backoff with jitter— service owner