SEV-1 INVESTIGATINGINC-1042

Duration: 14m 28s (MTTR reduced to 1m 24s)

Payment API Latency Spike & 5xx Outage

Service: payment-apiCluster: prod-us-east-2Commander: Sarah Chen (Lead SRE)Started: 10:24 AM

DB CONNECTIONS

100 / 100

Full Ceiling (Blocked)

P95 LATENCY

4.82 s

+1,908% Spike

HTTP 5XX RATE

31.4%

Checkout Failing

THROUGHPUT

1.24k req/s

Checkout Ingress

AI Root Cause Analysis (Gemini 2.5 Flash + Pinecone)

98% Confidence

Isolated Root Cause: Deployment v1.8.+2 introduced an unclosed DB cursor in async worker (checkout_worker.py:L142), leaking active connections until the 100/100 pool limit was saturated.

PG Pool

100/100 slots

Loki Errors

4,821 timeouts

Git Commit

8b7f3a1

Pinecone Match

#rb-db-pool-01

Recommended Action & Human Gate

Authorization Required

Rollback payment-api v1.8.2 → v1.8.1

Impact: 12 pod replicas • Recovery: < 45s • Schema Risk: Zero migrations

Real-Time Network Stream (Ports 4000 & 8000)Live HTTP requests from browser

Waiting for network transactions...

AI SRE Copilot

Gemini 2.5 Flash • Port 8000 (Live)

Quick Questions: