[Eval][v2][en] Third-party payment gateway timeouts cascade into checkout outage

Summary

Severity reasoning: User-facing outage: checkout success rate dropped from 99.5% to 22% (>1% error rate for >5 min). Revenue path broken: estimated $180k GMV lost. Regional unavailability: Stripe Connect latency spike affects us-east-1. This meets SEV1 criteria.

deepseek-chat·prompt v2·output: en·13320ms·1751↑ / 2078↓ tok·$0.00276

Root cause hypotheses

• highStripe Connect API latency spike due to upstream issue in us-east-1

  Evidence: Stripe status page: 'Investigating elevated latency for Connect endpoints in us-east-1' posted 18:35 UTC. APM shows p99 latency for /v1/payment_intents jumped from 800ms to 27s.

• highThread pool exhaustion caused by all worker threads blocking on Stripe calls

  Evidence: Thread pool utilization at 100% (200 threads), inbound queue depth 4200 (limit 5000). No circuit breaker or bulkhead on Stripe client.

• mediumStripe call timeout (30s) matches inbound gateway timeout, causing cascading failures

  Evidence: p99 latency 28s, hitting 30s timeout. Timeout configuration creates a tight coupling between downstream and upstream timeouts.

• lowResource leak in Stripe HTTP client (e.g., connection pool exhaustion)

  Evidence: No direct evidence yet, but thread pool exhaustion could be exacerbated by connection pool starvation. Check connection pool metrics.

• lowNetwork issue between our cluster and Stripe API (e.g., DNS, firewall, TLS)

  Evidence: No evidence of network errors in logs; latency is high but connections are succeeding. Stripe status page confirms their issue.

Investigation checklist

1. Check Stripe status page for updates

   curl -s https://status.stripe.com/api/v2/status.json | jq '.status.description'

   Copy

   Expected: Should show 'Investigating' or 'Identified' for Connect endpoints

2. Verify Stripe API latency from our cluster

   kubectl exec -n prod deploy/checkout-svc -- curl -o /dev/null -s -w 'time_total: %{time_total}\n' -X POST https://api.stripe.com/v1/payment_intents -H 'Authorization: Bearer <redacted>' -d 'amount=100¤cy=usd' --connect-timeout 5 --max-time 30

   Copy

   Expected: time_total should be <1s normally; if >5s, confirms Stripe latency

3. Check thread pool and queue depth metrics

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/metrics | grep -E 'thread_pool|queue_depth'

   Copy

   Expected: thread_pool_active == 200 (max), queue_depth near 5000

4. Check Stripe HTTP client connection pool metrics

   kubectl exec -n prod deploy/checkout-svc -- curl -s localhost:8080/metrics | grep -E 'stripe_connections|http_client'

   Copy

   Expected: If connection pool exhausted, active connections == max

5. Check for any recent deployments or config changes

   kubectl rollout history deploy/checkout-svc -n prod

   Copy

   Expected: No recent changes; if changes exist, check for timeout or thread pool config modifications

6. Check downstream dependencies (auth-svc, fraud-svc) for any anomalies

   kubectl logs -n prod -l app=auth-svc --since=10m | grep -i error | tail -5; kubectl logs -n prod -l app=fraud-svc --since=10m | grep -i error | tail -5

   Copy

   Expected: No errors; these services are normal

Mitigation plan

• Enable circuit breaker on Stripe client to fail fast when latency exceeds threshold (e.g., 5s). This will protect thread pool from being blocked.

  Risk: Circuit breaker may trip prematurely if threshold too low, causing false positives. Safer than disabling Stripe entirely.

  Rollback: Disable circuit breaker by reverting config change or setting threshold to 30s (original timeout).

• Implement bulkhead isolation for Stripe calls: dedicate a separate thread pool (e.g., 20 threads) for Stripe calls to prevent exhausting main worker pool.

  Risk: If Stripe calls exceed bulkhead capacity, they will be rejected, but main pool remains healthy. Requires code change; may not be immediate.

  Rollback: Revert bulkhead config to use shared pool.

• Reduce Stripe call timeout from 30s to 5s to fail fast and free threads sooner. This reduces thread blocking duration.

  Risk: Some legitimate Stripe calls may timeout if latency is high but not failing. Trade-off between availability and success rate.

  Rollback: Revert timeout to 30s.

• If Stripe issue persists and impact is unacceptable, disable Stripe payment path and fall back to an alternative payment method (e.g., PayPal) or show maintenance page.

  Risk: Lost revenue from Stripe payments; customer frustration. Destructive operation: must ensure fallback is tested.

  Rollback: Re-enable Stripe path by reverting feature flag or config.

Customer impact

Approximately 78% of checkout attempts are failing with a 504 error. In the last 5 minutes, about 3,000 customers were unable to complete purchases. Estimated lost revenue is $180,000 and growing. Customers see a timeout error page after 30 seconds.

Postmortem draft

Follow-ups

• P0Implement circuit breaker for Stripe client with 5s timeout threshold— service owner
• P0Implement bulkhead isolation for Stripe calls with dedicated thread pool— service owner
• P1Reduce Stripe call timeout from 30s to 5s— service owner
• P1Add monitoring for thread pool utilization and queue depth with alerts— platform team
• P2Review and update incident response runbook for Stripe-related outages— on-call SRE
• P2Evaluate fallback payment methods for Stripe unavailability— product team