Demo Mode
Aug 11, 2026 11:14:38
Ongoing
Checkout Service began returning HTTP 503 from all three configured probe regions within 95 seconds of each other, following a four-minute monotonic rise in time-to-first-byte from a 190 ms baseline to the upstream proxy's 5,000 ms ceiling. DNS resolution, TCP connect and TLS handshake times stayed within one standard deviation of baseline in every region throughout, which places the fault at the origin application rather than in the network path or at the edge. The ramp shape — progressive queueing rather than an abrupt step — is characteristic of resource exhaustion under an unchanged request rate. A discrete 28% shift in the TTFB baseline 41 minutes before the first failure, which never recovered, is consistent with a change landing on this service; Pingara has no deployment visibility and cannot confirm that directly.
Generated 20 minutes ago · gemini-2.5-flash
10:33:38
TTFB baseline shifted from 190 ms to 243 ms (+28%) and did not return. No errors, no status-code change.
11:10:08(us-east-1)
TTFB 480 ms — 2.5x the pre-shift baseline. Check still passing.
11:11:48(us-east-1)
TTFB 1,240 ms.
11:12:28(eu-west-1)
TTFB 1,090 ms — ramp now visible in a second region.
11:13:33(us-east-1)
TTFB 2,900 ms; response still HTTP 200.
11:13:58(us-east-1)
First HTTP 503. totalDuration 5,004 ms, ttfb 5,001 ms.
11:14:28(us-east-1)
Second consecutive failure; region marked failing.
11:14:38(eu-west-1)
First HTTP 503 in eu-west-1. Quorum reached (2 of 3 regions) — incident opened.
11:16:03(us-west-2)
First HTTP 503 in us-west-2. All three regions failing.
11:18:50
Acknowledged by Priya Raman.
11:25:38
DNS 18 ms, TCP connect 31 ms, TLS handshake 44 ms — all within baseline in all three regions. Network path is healthy.
11:41:08
Ongoing. No recovery observed; every check since T has returned HTTP 503 at the 5,000 ms ceiling.
Origin application saturation, not a network or edge fault
High confidenceAcross all three regions and the full incident window, DNS resolution (17–19 ms), TCP connect (29–34 ms) and TLS handshake (42–47 ms) remained within one standard deviation of their 7-day baselines, while TTFB rose by a factor of 26 before failing. The connection is established normally and the TLS session completes; the origin simply does not produce a first byte. That isolates the fault to the application tier behind the proxy.
Resource exhaustion (connection-pool or worker saturation) rather than a crash or restart
Medium confidenceThe pre-failure window shows a monotonic TTFB ramp — 190 → 480 → 1,240 → 2,900 ms over roughly four minutes — rather than a step change. A process crash, OOM kill or restart produces an abrupt transition from healthy to connection-refused; progressive queueing produces a ramp. The terminal state is HTTP 503 with a well-formed response body (responseSize 214 bytes, consistent across regions), which is a proxy or application rejecting requests under load, not an absent listener. Confidence is medium rather than high because Pingara observes only response timing; it cannot distinguish a saturated connection pool from a saturated thread pool or a downstream dependency queue.
Change-induced, landing approximately 41 minutes before the first failure
Medium confidenceTTFB baseline moved from 190 ms to 243 ms at a discrete point (T-41m), simultaneously in all three regions, and never recovered. A simultaneous cross-region shift with no corresponding change in DNS, TCP or TLS timing is a property of the origin, not of any one network path, and a step-and-hold shift is the signature of a deployment rather than of load. Pingara has no deployment visibility; this should be checked against the service's change log before it is treated as established.
Regional or cloud-provider fault (considered and argued against)
Low confidenceEvidence against: all three regions failed within 95 seconds of one another with identical timing signatures (ttfb 5,001 ± 6 ms) and an identical status code. A provider-side fault would be expected to affect regions non-simultaneously and with divergent timings, and the three probe regions do not share a failure domain. The correlated, identical failure points at the shared origin rather than at three independent networks.
- ·Checkout Service has no slowResponseThresholdMs configured, so detection fell back to apdexThreshold × 4 (2,000 ms) for 3 consecutive checks. The four-minute latency ramp therefore produced no degraded transition before total failure — the first signal anyone received was a complete outage.
- ·All three probe regions resolve to a single origin. This made quorum confirmation fast (a detection strength) but meant no region was serving successfully (an availability weakness). Regional detection breadth is not regional redundancy.
- ·The observed 5,000 ms ceiling is the upstream proxy's timeout, not Pingara's (the monitor timeout is 10,000 ms). True origin latency beyond five seconds is not observable from this monitor, so the ramp's actual endpoint is unknown.
- ·At a 99.95% monthly SLA target, Checkout Service's error budget for the period is 21.6 minutes. This incident has consumed approximately 26 minutes and is ongoing — the budget for the month is already exhausted.
Confirm whether a change landed on Checkout Service around T-41m. If so, roll back and re-measure TTFB against the 190 ms pre-shift baseline — recovery of the baseline confirms cause 3 and closes the incident.
Set slowResponseThresholdMs to 1,500 ms with 3 consecutive checks on this monitor. Against the observed ramp this would have opened a degraded state at approximately T-2m30s, roughly two and a half minutes before the first hard failure.
Add a second monitor against the checkout origin that bypasses the edge proxy, so origin latency is observable past the proxy's 5,000 ms ceiling. The current monitor cannot distinguish a 6-second origin from a 60-second one.
Storefront API shares this origin's region set and shows a smaller TTFB baseline shift in the same window. Check whether the change affected both services before scoping the rollback.
Define an explicit error-budget policy for Checkout Service. At 99.95% the monthly budget is 21.6 minutes; this single incident consumed it. Either the target reflects what the service can deliver, or the target is aspirational and the alerting attached to it will keep firing.
Aug 11, 2026 11:14:38
Aug 11, 2026 11:14:38
Aug 11, 2026 11:14:43
Aug 11, 2026 11:14:46
Aug 11, 2026 11:18:50
Aug 11, 2026 11:19:08
Aug 11, 2026 11:21:18
| Status | Time | Region | DNS | TCP | TLS | TTFB | Total | Error |
|---|---|---|---|---|---|---|---|---|
| 11:10:08 | us-east-1 | 18ms | 31ms | 44ms | 480ms | 573ms | - | |
| 11:11:48 | us-east-1 | 18ms | 31ms | 44ms | 1240ms | 1333ms | - | |
| 11:12:28 | eu-west-1 | 18ms | 31ms | 44ms | 1090ms | 1183ms | - | |
| 11:13:33 | us-east-1 | 18ms | 31ms | 44ms | 2900ms | 2993ms | - | |
| 11:13:58 | us-east-1 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 | |
| 11:14:28 | us-east-1 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 | |
| 11:14:38 | eu-west-1 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 | |
| 11:16:03 | us-west-2 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 | |
| 11:25:38 | us-east-1 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 | |
| 11:25:43 | eu-west-1 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 | |
| 11:25:48 | us-west-2 | 18ms | 31ms | 44ms | 5001ms | 5004ms | HTTP 503 |