Understanding Incidents

Incidents are the core of Pingara's alerting system. An incident represents a confirmed outage or service disruption, not a single failed check, but a verified problem that requires attention.

How Incidents Are Created

Pingara uses a consecutive failure model to prevent false positives from transient network blips.

The 2-Failure Rule

A single failed check does not create an incident. Instead:

1. First failure - Check fails, but no incident yet. The consecutive failure counter increments to 1.
2. Second consecutive failure - Check fails again. Counter reaches 2. Incident is created.

This ensures you're only alerted for real problems, not momentary network hiccups.

Check 1: ✅ Pass     → Status: Up
Check 2: ❌ Fail     → Status: Up (1 consecutive failure)
Check 3: ❌ Fail     → Status: Down (2 consecutive failures → Incident created!)
Check 4: ❌ Fail     → Status: Down (incident ongoing)
Check 5: ✅ Pass     → Status: Down (1 consecutive success)
Check 6: ✅ Pass     → Status: Up (2 consecutive successes → Incident resolved!)

What Counts as a Failure

A check is marked as "failed" if any of these occur:

• Unexpected status code - Response code not in the monitor's expected list. Every HTTP monitor created from the dashboard is stored with 200, 201, 202, 203, 204, 301, 302. There is no field to change it; a monitor created without the field at all (a path no UI or API takes today) would get the platform-wide default instead, 200-399
• Timeout - Server didn't respond within the configured timeout
• Connection error - DNS failure, TCP connection refused, TLS handshake failure
• Keyword check failed - the expected text is missing (or, in must NOT contain mode, present). This counts only for a monitor whose keyword-failure severity is set to down. New monitors default to degraded instead, which doesn't count toward this failure counter. It feeds the separate degraded counter, and opens a degraded incident rather than a down one. See HTTP/HTTPS Monitoring for both behaviors.

Multi-Region Monitoring and Quorum

When you monitor from multiple regions, Pingara applies a quorum rule to further reduce false positives.

How Quorum Works

Instead of trusting a single region's result, Pingara requires failures from multiple regions before confirming a problem. The rule is half your enabled regions, rounded up, with a floor of two. A single region is the one exception, since there's no one else for it to agree with:

A region only casts a down vote after 2 consecutive failed checks of its own (or 2 consecutive successes, to vote recovered). Those two thresholds are hard-coded, not settings you can change. The degraded vote is the exception: it uses the monitor's own slow-response consecutive-check count, which defaults to 3 and is configurable per monitor. A region whose most recent check is older than 5× the monitor's interval is also excluded from the vote entirely, rather than counted as either up or down, so a probe that's gone quiet can't drag a monitor's status either way.

Why Quorum Matters

Without quorum (single region):

US East check fails (ISP routing issue) → False incident!

With quorum (3 regions):

US East: ❌ Fail
EU West: ✅ Pass
AP South: ✅ Pass
→ 1 of 3 failed → NOT an incident (quorum of 2 not reached)

Actual outage (4 regions):

US East:  ❌ Fail
US West:  ❌ Fail
EU West:  ✅ Pass
AP South: ✅ Pass
→ 2 of 4 failed → Incident created (quorum reached, the other two
  passing doesn't save it)

Recommendation: Enable at least 2 regions for production monitors. Use 3-4 regions for critical services.

Incident Lifecycle

Every incident progresses through a defined set of statuses:

1. Investigating

The state a down incident opens in.

• Trigger: 2 consecutive failures per region, quorum-confirmed
• Actions: Alerts sent to all configured channels
• What it means: Pingara has detected a problem and is tracking it

2. Identified

The cause of the incident has been identified.

• Trigger: A manual status update, and only that. Generating AI root cause hints saves the hints onto the incident but never changes its status. You move it to Identified yourself once you've read them
• What it means: The team understands the problem

3. Monitoring

A fix has been applied and the team is watching for confirmation. It is also where a degraded incident starts: the slow-response rule opens its incident directly at Monitoring rather than at Investigating, because there is nothing failing to investigate.

• Trigger: Manual status update, or a degraded incident being opened
• What it means: Recovery is expected, watching closely

4. Resolved

The incident is over. Service has been restored.

• Trigger: 2 consecutive successful checks (automatic) or manual resolution
• Actions: Recovery alerts sent to all configured channels on automatic resolution only. They aren't sent a second time when Pingara already sent a recovery notice for the incident and hasn't alerted about it since (see Reopening an incident below). Manual resolution sends no notification (see Resolving an Incident below).
• What it means: The problem is fixed

Investigating → Identified → Monitoring → Resolved
     ↑              ↑            ↑            ↑
  (automatic)   (manual)    (manual)    (automatic)

Automatic Resolution

Pingara resolves incidents automatically using the same consecutive model as creation:

The 2-Success Rule

1. First success - Check passes, but incident stays open. Consecutive success counter increments to 1.
2. Second consecutive success - Check passes again. Counter reaches 2. Incident is resolved.

This prevents premature resolution from a single lucky check during an intermittent outage.

What Happens on Automatic Resolution

1. Incident status changes to Resolved
2. resolvedAt timestamp is recorded
3. Recovery alert notifications are dispatched, unless Pingara already sent one for this incident and hasn't alerted about it since (see Reopening an incident below)
4. Monitor status returns to Up
5. Consecutive failure counter resets to 0

You can also resolve an incident manually from the incident detail page. Manual resolution makes the same status change, but it does not send a recovery notification. Recovery alerts and status-page subscriber updates go out only when the probe fleet itself resolves the incident, normally after two consecutive successful checks, as described above. If you fix an outage out-of-band and resolve the incident manually, tell your on-call channel and status-page subscribers yourself; Pingara won't. The stale "down" state on a status page stays until the monitor recovers on its own, which never happens if it's paused or its target has changed. What manual resolution does record is who did it. See below for details.

Incident Details

Each incident records comprehensive information for post-mortems:

Core Data

• Monitor - Which monitor detected the issue
• Status - Current lifecycle stage
• Started at - When the incident was created
• Resolved at - When recovery was confirmed
• Duration - Total time from start to resolution

Diagnostic Data

• Error type - Category of failure (timeout, DNS failure, connection refused, etc.)
• Error message - Detailed error description
• Affected regions - Which monitoring regions detected the failure
• Average response time - Mean response time during the incident

AI Analysis

• Root cause hint - AI-generated suggestions for what likely caused the incident (see Root Cause Analysis)

Viewing Incidents

Incident List

Navigate to Incidents in the sidebar to see all incidents across your organization.

Filters available:

• Status - Investigating, Identified, Monitoring, Resolved
• Monitor - Filter by specific monitor
• Time range - Last 24h, 7d, 30d, or custom

Incident Detail

Click any incident to see a full timeline, built from every recorded event plus notification delivery records:

• Incident started - when detection confirmed the outage
• Acknowledged - who acknowledged it, or a note that the actor wasn't recorded (see below)
• Status changed - each move through the lifecycle. A resolution names the resolver, says "Resolved automatically" for a system recovery, or notes that the actor wasn't recorded for an incident resolved before this feature shipped (see below)
• Notes and root cause hint updated - when either field changes, and who changed it
• RCA report generated - automatically after resolution, or on demand
• Escalation notices - each repeat page, numbered against the policy's cap
• Alert delivery - sent and failed notifications, with the channel and failure reason

Alongside the timeline: performance metrics at time of failure, affected regions, and root cause hints (AI-generated).

Acknowledging an Incident

Acknowledging tells your team "someone has seen this and is on it." Any member of your organization can acknowledge an incident, including a viewer, because the person who notices an outage first is often not the one with admin access, and getting them to hand off before the paging stops costs time you don't have during an outage.

What acknowledging does:

• Records who acknowledged the incident and when
• Stops escalation - see Setting Up Alerts for how escalation chains work
• Does not change the incident's status. An acknowledged incident can still be Investigating, Identified, or Monitoring

First acknowledger wins. Once an incident is acknowledged, acknowledging it again has no effect. It doesn't reassign credit or add a second timeline entry. This matters because acknowledgement is a "stop paging" signal, not a claim you can hand off by clicking again.

Incidents acknowledged before this feature shipped show "Acknowledged" in the timeline with a note that who did it wasn't recorded at the time. Pingara never guesses or backfills an actor, because a wrong attribution on an incident record is worse than an absent one.

Resolving an Incident

An incident resolves one of two ways: automatically, when two consecutive checks pass (see Automatic Resolution above), or manually, when someone on your team sets its status to Resolved. Both close the incident and record who or what resolved it, but only automatic resolution sends a recovery notification. Resolving manually does not alert your team or update status-page subscribers. See Automatic Resolution above for what to tell them yourself.

You'll see one of three things on the incident detail page and in the Status column of your incidents list:

• "Resolved by <name>" - a team member resolved it. The list shows their name next to a user icon; the detail page names them the same way it names who acknowledged the incident.
• "Resolved automatically" - the probe fleet observed two consecutive successful checks and closed the incident with nobody involved. The list shows a bot icon and "Auto".
• Neither - the incident resolved before this feature shipped. Pingara doesn't know whether a person resolved it or it recovered on its own, and says so rather than guessing. This is different from "Resolved automatically": the platform genuinely never recorded which one happened, so it doesn't claim either.

If you see a colleague's name on a recent incident and a blank on an older one, that's expected. It's not a bug, and it's not a permissions issue. Pingara only started recording who resolved an incident as of this release; anything resolved earlier simply predates that record, the same way incidents acknowledged before the acknowledgement feature shipped show no acknowledger.

First resolution wins, exactly like acknowledgement: once an incident moves to Resolved, submitting "Resolved" again, for example, by resubmitting a stale form, doesn't move the resolution time or reassign credit to whoever submitted it. This also keeps your Mean Time to Resolve accurate, since MTTR is calculated from the moment an incident first resolves (see below).

Reopening an incident

Reopening an incident doesn't notify your alert channels. Setting a resolved incident back to Investigating, Identified or Monitoring sends nothing to your alert channels or status-page subscribers, whether you do it in the app or with PATCH /api/v1/incidents/{id}. If the monitor is up when you reopen it, Pingara resolves the incident again automatically on a following check, as long as the monitor stays up. Whether that resolution sends a recovery notice depends on what your channels were last told:

• If Pingara already sent a recovery notice for this incident and hasn't alerted about it since, it doesn't send another. Your channels were told it recovered and weren't told it reopened, so a second "recovered" would report a change they never saw. The incident timeline says no recovery notice was sent.
• If Pingara hasn't sent a recovery notice for this incident, for example because you resolved it by hand, the recovery notice is sent. A manual resolve sends nothing, so your channels' last word on the incident is still the down or degraded alert; the automatic resolution closes it, including resolving the PagerDuty incident that alert opened.
• If your channels are alerted about the incident again while it's reopened, because the monitor goes down or an escalation notice goes out, the later recovery notice is sent too.

Webhook subscriptions are separate: when Pingara holds back a repeat recovery notice, incident.resolved is still sent for the automatic resolution, because your subscription normally received an event for the reopen. A manual resolve or reopen's webhook emit is also subject to its own per-incident and per-organization hourly caps, so a busy incident or organization can occasionally miss one even when nothing above withholds it. See Webhook subscriptions for the full event shape and those caps.

Response Time Metrics

The Incidents page and your dashboard show two response metrics, computed from your organization's incident history:

• MTTA (Mean Time to Acknowledge) - the average time between an incident starting and someone acknowledging it
• MTTR (Mean Time to Resolve) - the average time between an incident starting and it resolving

MTTR is measured from detection, not from acknowledgement. Both conventions exist across the industry; Pingara measures from the moment a check confirms the outage, so MTTR always includes any time the incident sat unnoticed before someone acknowledged it. The Response Time card's tooltip states this explicitly.

A few rules keep these numbers honest rather than flattering:

• Test incidents are excluded. Incidents you trigger with the "simulate failure" tool auto-resolve in about a minute and would otherwise drag MTTR toward that number.
• Coverage ships with the average. An incident that was never acknowledged doesn't count toward MTTA, but it still counts in the "X of Y acknowledged" line shown next to it, so an organization that acknowledges 1 of 50 incidents can't show a misleadingly fast MTTA with nothing to indicate the gap. Unresolved incidents work the same way for MTTR.
• You'll see a dash, never a zero, when there's nothing to measure. If nothing has been acknowledged yet, MTTA shows "–" rather than "0". A zero would read as "we acknowledge instantly," which is the opposite of the truth when the real answer is "nobody has acknowledged anything yet."
• A single incident is labeled as such, so you don't mistake one data point for a trend.

These metrics are computed live from your incident history. They're not part of SLA reports, the Developer API, or the weekly email digest today.

Incident vs Degraded

It's important to understand the difference:

A monitor that's already Degraded and then fails outright doesn't open a second incident. The existing degraded incident is promoted in place to Down, keeping the same incident and the same ID, and Pingara sends a real down alert on every channel, the same as it would for a brand-new down incident. That means a monitor which goes Degraded and later Down produces two events on that one incident, not one: incident.degraded when it opened, then incident.created when it's promoted.

Best Practices

Set Appropriate Intervals

For critical services, use shorter check intervals:

• 30 seconds - Fastest detection (~1 minute to incident)
• 1 minute - Good balance (~2 minutes to incident)
• 5 minutes - Standard monitoring (~10 minutes to incident)

Enable Multi-Region

Single-region monitoring leads to false positives. Always use at least 2 regions for production monitors.

Review Incidents Regularly

Schedule weekly or monthly incident reviews:

1. How many incidents occurred?
2. What was the average resolution time?
3. Were there any false positives?
4. Should Apdex thresholds be adjusted?

Use the AI root cause hints

Pingara's AI-generated root cause hints can suggest why an incident occurred, not just that it happened. See Root Cause Analysis for details.

Next Steps

• Root Cause Analysis - AI-generated hints for what caused an incident
• Setting Up Alerts - Get notified when incidents occur
• HTTP/HTTPS Monitoring - Configure your monitors