Alerts

Overview — what it does

Infrastructure ▸ Alerts (/infra/alert) is the alert-group (policy) console. An operator adds conditions across four areas (system / web / DB / security) to a single policy (infr_alert_policy), tunes each threshold factor on screen, and when a breach occurs the group's recipients (infr_alert_recipient) are notified by SMS/email automatically.

It replaces the old "Capacity & Alerts" page (/infra/capacity), which now redirects here.

Key characteristics:

• A policy is one monitoring bundle: severity, re-alert suppression (cooldown), areas, conditions, channels, and recipients all live on one card. Create several policies split by team or shift (e.g. "Infra night on-call", "Security ops").
• Evaluation and dispatch are separated: AlertEvaluator evaluates conditions; AlertDispatcher opens alerts and sends. Both are shared by the console [Check now] button and the scheduled command infr:eval-alerts.
• Dedup, cooldown, batch, auto-close: repeated breaches collapse into one alert, re-sends are suppressed during cooldown, a policy's notifications are batched into one message (prevents SMS floods), and alerts auto-close when the breach clears.
• Read-only live data: system from server_status, DB from infr_db_metric, security from legacy_glory detection tables, web from a live HTTP probe. Web cert-expiry scans infr_web_cert globally.

This screen manages only policies, alerts, and delivery history. For per-server graphs see Servers; for web uptime see Service ▸ Web; for DB load see Service ▸ Database. Threshold defaults are defined in Operations Policy.

When to use

• You want immediate notice when a server's load average or free disk crosses a danger line.
• You want on-call notified when a landing URL is down (unexpected status / no response) or responds slowly.
• You want continuous watch for failed TLS auto-renewal on registered web services (cert about to expire).
• You want alerts when a registered DB's pending queue backs up or hourly slow-query count spikes.
• You want to broadcast breach signals to the team: IDS alert floods, AI anomaly-block surges, new blocked-IP surges.

Layout

Policy card grid

Each policy renders as one card summarizing its active toggle, severity badge, enabled areas, and recipient count, with these actions:

Edit modal (areas, conditions, channels, recipients)

• Basic: name, severity (info/warn/crit), re-alert suppression (minutes), active, note.
• Alert areas: enable system / web / DB / security (system on by default).
• Conditions: per enabled area, on/off + threshold factor (see condition catalog below).
• Channels: SMS and mail. Actual delivery requires a provider in Mail Adapters/SMS Adapters; the card warns when no provider is configured.
• Recipients (repeater): name, email, phone + via_mail/via_sms checks. The effective channel is the intersection of policy channels ∩ recipient via_*.

Recent alerts feed

Shows infr_alert ordered by last_seen desc, up to 60 rows (30s polling). Each row shows the observed value, threshold, aggregate count, and sent ok/total (from infr_alert_delivery), with [Ack] and [Close] status transitions.

How to use

1) Create a policy

1. [New alert group] → enter name, severity, cooldown.
2. Enable the areas you want to watch.
3. For each area, enable conditions and adjust threshold factors. Defaults are prefilled from Operations Policy alertDefaults().
4. Pick channels (SMS/mail) and add recipients.
5. Save.

2) Condition catalog

System (server_status, seq window)

• Target defaults to all servers (targets.system.all=true); you can narrow to selected aliases (max 200).
• The snapshot reads only the latest seq window (distinct seq desc limit 3) to avoid slow queries (inherited from HealthService). Offline / no-data servers are not judged — a server must have loadavg in the latest 2 cycles to be online.
• Free disk uses the tightest partition among df-, df-data, df-home (DfParse::availableGb).

Web (live HTTP probe · infr_web_cert)

• Target URLs come from targets.web.urls[] (with labels, max 30 per policy). Each evaluation cycle requests them directly via Http::connectTimeout(4)->withoutVerifying()->get(). A status different from expected, or a connection failure/timeout, is treated as service OFF (when the code condition is on).
• Cert-expiry is a single toggle but the scan is global: independent of the policy's target URLs, it sweeps the infr_web_cert records for every domain of every registered web service and raises a breach for each domain expiring within the threshold (web_cert_hours, default 48h) — up to 50 per evaluation. Certificates are refreshed on a 6-hour rotation by WebProbeService. This catches a failed auto-renewal before an expiry surprises anyone.

Enabling just "Cert expiry imminent" watches every site's certificate. It is not per-site — it is a global scan across all domains.

Database (infr_db_metric latest sample)

• Targets are targets.database.collectors[] (access=database collector keys). If no target is selected, nothing is evaluated.
• It reads DbCollectorService::latest($collector) but only judges samples that are ok and under 10 minutes old (avoids stale false positives). Register/collect via DB Collectors.

Security (legacy_glory detection tables · composite)

The evaluation window is sec_window_min (default 10 min). Any one signal over its threshold fires a breach.

Related: Threats · Detections · Blocklist.

3) Check now / Test

• [Check now] (run): runs evaluate() → handle() for this one policy immediately and returns a breach/sent summary (breaches :b · sent :s) plus the refreshed feed. Use it to confirm a condition actually fires without waiting for the scheduler.
• [Test] (test): sends a test mail/SMS to active recipients right now to confirm the channel works (independent of any breach — a provider health check).

4) Automatic evaluation (schedule)

Active policies are evaluated and dispatched automatically every minute.

Schedule::command('infr:eval-alerts --quiet-if-ok')
    ->everyMinute()
    ->withoutOverlapping(2)
    ->onOneServer()
    ->when($is_leader);

Manual runs:

# Evaluate all active policies (single-shot, exits after — not a daemon)
php artisan infr:eval-alerts

# A single policy only
php artisan infr:eval-alerts --policy=12

# Suppress output when there is no breach (scheduler default)
php artisan infr:eval-alerts --quiet-if-ok

infr:eval-alerts is a single-shot command that exits (it loops the policies and returns). Never turn it into an infinite-loop daemon — spawning a never-exiting daemon from the scheduler exhausts shared DB connections (see the pulse:check connection-leak lesson). The scheduler safely launches a fresh instance every minute.

Data sources

Write tables (new infr_)

Read sources (legacy + new, read-only)

• System: server_status (legacy connection, part=loadavg/df-*).
• Web: live HTTP GET (target URLs) + infr_web_cert (cert expiry).
• DB: infr_db_metric (populated by access=database collectors).
• Security: suricata_alert_ct, kitsune_detail_log, blacklist_info on legacy_glory.

Dedup / cooldown / auto-close flow

1. AlertEvaluator::evaluate($policy) → array of breach candidates per active area. Area exceptions are isolated (one area failing does not block the rest).
2. AlertDispatcher::handle($policy, $breaches):
   • Each breach is looked up by (policy_id, metric, entity_key, status=open) → if found, update + count++; else insert new (first_seen, count=1, notified_at=null).
   • Notifiable = new OR notified_at null OR notified_at <= now - cooldown_min.
   • Auto-close: open alerts of this policy not present in this round's breaches → status='closed'.
   • Batch send: notifiable alerts are bundled into one message per recipient per channel (body max 8 lines, overflow "+N more"; SMS capped at 140 chars). After sending, each target's notified_at=now.
   • One infr_alert_delivery row per (recipient, channel).

Access

Level scale: demo=1 · staff=5 · admin=10. Reads require level≥1; writes and dispatch require level≥2. All destructive/dispatch operations are recorded in infr_audit_log (see Audit Log). Delivery-channel providers are configured by an admin (level≥10).

Troubleshooting & FAQ

Q. I enabled a condition but no alert arrives.

1. Confirm the policy is active and the scheduler (schedule:work / prod supervisor) is running.
2. No channel provider — check the card warning and configure a default provider in Mail Adapters/SMS Adapters.
3. Verify the intersection of recipient via_mail/via_sms with the policy channels is non-empty, and that email/phone are filled.
4. During cooldown, re-sends are suppressed (cooldown_min, default 30 min). Use [Check now] to verify immediately, but an already-notified alert only re-sends after cooldown passes.

Q. My DB/system conditions stay quiet.

• System: the target server must be online (loadavg in the latest 2 seq cycles) to be judged. Offline/no-data is skipped.
• DB: the infr_db_metric latest sample must be ok and under 10 minutes old. Confirm the collector is running in DB Collectors.

Q. Does cert-expiry watch only a specific site?

• No. web_cert_on is a global scan — it sweeps infr_web_cert for every domain of every registered web service, independent of the policy's target URLs.

Q. Does the same breach keep piling up?

• No. It aggregates onto one open alert keyed by (policy_id, metric, entity_key) (count++). When the breach clears, the next evaluation auto-closeds it.

Q. Does the web probe slow down evaluation?

• URLs are capped at 30 per policy with a 4s connect timeout and a short latency-based timeout. The system snapshot reads only the seq window to avoid slow queries.

Related

• Operations Policy — defines threshold defaults (alertDefaults())
• Mail Adapters · SMS Adapters — delivery-channel providers
• Service ▸ Web — cert/uptime source (infr_web_cert)
• DB Collectors — DB condition source (infr_db_metric)
• Servers · Health — system condition source (server_status)
• Threats · Blocklist — security condition sources