Understanding Incidents

Incidents are the core of Pingara's alerting system. An incident represents a confirmed outage or service disruption — not a single failed check, but a verified problem that requires attention.

How Incidents Are Created

Pingara uses a consecutive failure model to prevent false positives from transient network blips.

The 2-Failure Rule

A single failed check does not create an incident. Instead:

1. First failure — Check fails, but no incident yet. The consecutive failure counter increments to 1.
2. Second consecutive failure — Check fails again. Counter reaches 2. Incident is created.

This ensures you're only alerted for real problems, not momentary network hiccups.

Check 1: ✅ Pass     → Status: Up
Check 2: ❌ Fail     → Status: Up (1 consecutive failure)
Check 3: ❌ Fail     → Status: Down (2 consecutive failures → Incident created!)
Check 4: ❌ Fail     → Status: Down (incident ongoing)
Check 5: ✅ Pass     → Status: Down (1 consecutive success)
Check 6: ✅ Pass     → Status: Up (2 consecutive successes → Incident resolved!)

What Counts as a Failure

A check is marked as "failed" if any of these occur:

• Unexpected status code — Response code not in the expected list (default: 200-204)
• Timeout — Server didn't respond within the configured timeout
• Connection error — DNS failure, TCP connection refused, TLS handshake failure
• Keyword missing — Response body doesn't contain the expected keyword (if keyword check is enabled)

Multi-Region Monitoring and Quorum

When you monitor from multiple regions, Pingara applies a quorum rule to further reduce false positives.

How Quorum Works

Instead of trusting a single region's result, Pingara requires failures from multiple regions before confirming a problem:

Why Quorum Matters

Without quorum (single region):

US East check fails (ISP routing issue) → False incident!

With quorum (3 regions):

US East: ❌ Fail
EU West: ✅ Pass
AP South: ✅ Pass
→ 1 of 3 failed → NOT an incident (regional issue)

Actual outage (3 regions):

US East: ❌ Fail
EU West: ❌ Fail
AP South: ❌ Fail
→ 3 of 3 failed → Incident created (confirmed outage)

Recommendation: Enable at least 2 regions for production monitors. Use 3-4 regions for critical services.

Incident Lifecycle

Every incident progresses through a defined set of statuses:

1. Investigating

The initial state when an incident is first created.

• Trigger: 2 consecutive failures confirmed
• Actions: Alerts sent to all configured channels
• What it means: Pingara has detected a problem and is tracking it

2. Identified

The cause of the incident has been identified (manually set or AI-assisted).

• Trigger: Manual status update or AI root cause analysis
• What it means: The team understands the problem

3. Monitoring

A fix has been applied and the team is watching for confirmation.

• Trigger: Manual status update
• What it means: Recovery is expected, watching closely

4. Resolved

The incident is over. Service has been restored.

• Trigger: 2 consecutive successful checks (automatic) or manual resolution
• Actions: Recovery alerts sent to all configured channels — automatic resolution only. Manual resolution sends no notification (see Resolving an Incident below).
• What it means: The problem is fixed

Investigating → Identified → Monitoring → Resolved
     ↑              ↑            ↑            ↑
  (automatic)   (manual)    (manual)    (automatic)

Automatic Resolution

Pingara resolves incidents automatically using the same consecutive model as creation:

The 2-Success Rule

1. First success — Check passes, but incident stays open. Consecutive success counter increments to 1.
2. Second consecutive success — Check passes again. Counter reaches 2. Incident is resolved.

This prevents premature resolution from a single lucky check during an intermittent outage.

What Happens on Automatic Resolution

1. Incident status changes to Resolved
2. resolvedAt timestamp is recorded
3. Recovery alert notifications are dispatched
4. Monitor status returns to Up
5. Consecutive failure counter resets to 0

You can also resolve an incident manually from the incident detail page. Manual resolution makes the same status change, but it does not send a recovery notification — recovery alerts and status-page subscriber updates go out only when the probe fleet itself observes two consecutive successful checks, as described above. If you fix an outage out-of-band and resolve the incident manually, tell your on-call channel and status-page subscribers yourself; Pingara won't. The stale "down" state on a status page stays until the monitor recovers on its own, which never happens if it's paused or its target has changed. What manual resolution does record is who did it — see below.

Incident Details

Each incident records comprehensive information for post-mortems:

Core Data

• Monitor — Which monitor detected the issue
• Status — Current lifecycle stage
• Started at — When the incident was created
• Resolved at — When recovery was confirmed
• Duration — Total time from start to resolution

Diagnostic Data

• Error type — Category of failure (timeout, DNS failure, connection refused, etc.)
• Error message — Detailed error description
• Affected regions — Which monitoring regions detected the failure
• Average response time — Mean response time during the incident

AI Analysis

• Root cause hint — AI-generated analysis of what likely caused the incident (see Root Cause Analysis)

Viewing Incidents

Incident List

Navigate to Incidents in the sidebar to see all incidents across your organization.

Filters available:

• Status — Investigating, Identified, Monitoring, Resolved
• Monitor — Filter by specific monitor
• Time range — Last 24h, 7d, 30d, or custom

Incident Detail

Click any incident to see a full timeline, built from every recorded event plus notification delivery records:

• Incident started — when detection confirmed the outage
• Acknowledged — who acknowledged it, or a note that the actor wasn't recorded (see below)
• Status changed — each move through the lifecycle. A resolution names the resolver, says "Resolved automatically" for a system recovery, or notes that the actor wasn't recorded for an incident resolved before this feature shipped (see below)
• Notes and root cause hint updated — when either field changes, and who changed it
• RCA report generated — automatically after resolution, or on demand
• Escalation notices — each repeat page, numbered against the policy's cap
• Alert delivery — sent and failed notifications, with the channel and failure reason

Alongside the timeline: performance metrics at time of failure, affected regions, and root cause hints (AI-generated).

Acknowledging an Incident

Acknowledging tells your team "someone has seen this and is on it." Any member of your organization can acknowledge an incident — including a viewer — because the person who notices an outage first is often not the one with admin access, and getting them to hand off before the paging stops costs time you don't have during an outage.

What acknowledging does:

• Records who acknowledged the incident and when
• Stops escalation — see Setting Up Alerts for how escalation chains work
• Does not change the incident's status — an acknowledged incident can still be Investigating, Identified, or Monitoring

First acknowledger wins. Once an incident is acknowledged, acknowledging it again has no effect — it doesn't reassign credit or add a second timeline entry. This matters because acknowledgement is a "stop paging" signal, not a claim you can hand off by clicking again.

Incidents acknowledged before this feature shipped show "Acknowledged" in the timeline with a note that who did it wasn't recorded at the time. Pingara never guesses or backfills an actor — a wrong attribution on an incident record is worse than an absent one.

Resolving an Incident

An incident resolves one of two ways: automatically, when two consecutive checks pass (see Automatic Resolution above), or manually, when someone on your team sets its status to Resolved. Both close the incident and record who or what resolved it, but only automatic resolution sends a recovery notification. Resolving manually does not alert your team or update status-page subscribers — see Automatic Resolution above for what to tell them yourself.

You'll see one of three things on the incident detail page and in the Status column of your incidents list:

• "Resolved by <name>" — a team member resolved it. The list shows their name next to a user icon; the detail page names them the same way it names who acknowledged the incident.
• "Resolved automatically" — the probe fleet observed two consecutive successful checks and closed the incident with nobody involved. The list shows a bot icon and "Auto".
• Neither — the incident resolved before this feature shipped. Pingara doesn't know whether a person resolved it or it recovered on its own, and says so rather than guessing. This is different from "Resolved automatically": the platform genuinely never recorded which one happened, so it doesn't claim either.

If you see a colleague's name on a recent incident and a blank on an older one, that's expected — it's not a bug, and it's not a permissions issue. Pingara only started recording who resolved an incident as of this release; anything resolved earlier simply predates that record, the same way incidents acknowledged before the acknowledgement feature shipped show no acknowledger.

First resolution wins, exactly like acknowledgement: once an incident moves to Resolved, submitting "Resolved" again — for example, by resubmitting a stale form — doesn't move the resolution time or reassign credit to whoever submitted it. This also keeps your Mean Time to Resolve accurate, since MTTR is calculated from the moment an incident first resolves (see below).

Response Time Metrics

The Incidents page and your dashboard show two response metrics, computed from your organization's incident history:

• MTTA (Mean Time to Acknowledge) — the average time between an incident starting and someone acknowledging it
• MTTR (Mean Time to Resolve) — the average time between an incident starting and it resolving

MTTR is measured from detection, not from acknowledgement. Both conventions exist across the industry; Pingara measures from the moment a check confirms the outage, so MTTR always includes any time the incident sat unnoticed before someone acknowledged it. The Response Time card's tooltip states this explicitly.

A few rules keep these numbers honest rather than flattering:

• Test incidents are excluded. Incidents you trigger with the "simulate failure" tool auto-resolve in about a minute and would otherwise drag MTTR toward that number.
• Coverage ships with the average. An incident that was never acknowledged doesn't count toward MTTA — but it still counts in the "X of Y acknowledged" line shown next to it, so an organization that acknowledges 1 of 50 incidents can't show a misleadingly fast MTTA with nothing to indicate the gap. Unresolved incidents work the same way for MTTR.
• You'll see a dash, never a zero, when there's nothing to measure. If nothing has been acknowledged yet, MTTA shows "–" rather than "0" — a zero would read as "we acknowledge instantly," which is the opposite of the truth when the real answer is "nobody has acknowledged anything yet."
• A single incident is labeled as such, so you don't mistake one data point for a trend.

These metrics are computed live from your incident history — they're not part of SLA reports, the Developer API, or the weekly email digest today.

Incident vs Degraded

It's important to understand the difference:

Best Practices

Set Appropriate Intervals

For critical services, use shorter check intervals:

• 30 seconds — Fastest detection (~1 minute to incident)
• 1 minute — Good balance (~2 minutes to incident)
• 5 minutes — Standard monitoring (~10 minutes to incident)

Enable Multi-Region

Single-region monitoring leads to false positives. Always use at least 2 regions for production monitors.

Review Incidents Regularly

Schedule weekly or monthly incident reviews:

1. How many incidents occurred?
2. What was the average resolution time?
3. Were there any false positives?
4. Should Apdex thresholds be adjusted?

Use Root Cause Analysis

Pingara's AI-powered root cause analysis can help you understand why an incident occurred, not just that it happened. See Root Cause Analysis for details.

Next Steps

• Root Cause Analysis — AI-powered incident diagnostics
• Setting Up Alerts — Get notified when incidents occur
• HTTP/HTTPS Monitoring — Configure your monitors