Alerts
This chapter lists when the service sends alerts and where it keeps their state. Setup is described in the Administration Guide (Failure and silence alerts).
Overview
| Property | Value |
|---|---|
| Evaluated by | The leader node, every minute (loop Report Alerts), from the finished report executions in AF |
| Failure | The newest counted run ends Failed or CompletedWithErrors |
| Missed schedule | An enabled report with a scheduled trigger has no run that ended Success or CompletedWithErrors triggered at or after the newest scheduled time that is at least AlertSilenceGraceMinutes old. Scheduled times before the report's last change, or before alerts were turned on for it, are not checked |
| First check | When alerts are turned on (globally or with SendAlerts), and for a state without ArmedUtc, the first check records the current state (Failing when the newest run failed, otherwise Ok) and sends nothing (ArmedUtc in AlertState). Turning alerts off clears ArmedUtc |
| Service outage | The time from the last heartbeat before a node's leadership term to the start of the term, when longer than 90 seconds (restart or failover). A missed scheduled time inside it becomes an Outage notice instead of a missed-schedule alert; a later run of such a report sends no recovery alert. The outage is known to the leader that started the term; after a further restart missed times count as ordinary missed schedules |
| Not counted | Previews (IsDryRun), backfills without delivery (SkipDelivery), declined runs, display prints |
| States and alerts | Ok to Failing or Silent to Failing: failure alert; Ok to Silent: missed-schedule alert; Failing or Silent to Ok after a successful run: recovery alert, only when the Failing or Silent state was announced (Announced); Ok to Silent during an outage: outage notice. No other change sends an alert |
| Recipients | Report owner (mail address from Active Directory) and AlertRecipients by mail; AlertTeamsWebhookUrls by Teams |
| Grouping | All alerts of one check go to a recipient in one mail (recipients with the same alerts get the same mail) and to each Teams webhook in one message; outage notices of the same outage form one section. At most 1 MB of screenshots per mail |
| Content | Report, run, errors, failed page and step from the failure evidence, a link that opens the run details in PI Vision (vrp-action=open-execution); the first evidence screenshot is attached to the mail when it is at most 300 KB |
| State | Report element attribute AlertState (template Report, AF migration 15), JSON. It is written before the alert is sent |
| Retries | An alert no destination accepted is retried every minute, up to 5 attempts |
| Report field | SendAlerts (Send alerts), report JSON, default true |
