vaspian-alerts
Vaspian operational alerts (incidents, degraded services)
Vaspian operational alerts — fraud detection on outbound calls and platform-health regressions on the voice infrastructure. Loud topic; treat any P1/P2 here as oncall-worthy.
ntfy.sh/vaspian-alerts · added 2026-05-13
Publishers
Every code path / config block that produces messages on this topic today. If you wire a new publisher, add it here.
Called by VoIPMonitor's external_script hook every time the native alert engine fires a rule. Takes positional args (alert_id, alert_name, unix_ts, JSON of triggering CDR IDs), queries MySQL to enrich the alert with the last-5m and 24h-ago metric snapshots, looks up the triggering CDRs, and fans out to whichever channels are enabled in /etc/vmon-alert/config.json — Brevo email, Microsoft Teams webhook, Slack webhook, and ntfy. Severity (P1/P2/P3) is derived from a substring match on alert name (see SEVERITY_RULES in the script) and drives both the ntfy Priority header and the tag list. 30-minute dedup state lives in /var/lib/voipmon-alert/state.json; the same alert re-firing inside that window is dropped silently.
- · P1 (priority 5, rotating_light + red_circle): Extreme PDD, No CDRs, PDD >10s, Intl premium-CC, Intl long-duration, Intl burst
- · P2 (priority 4, warning): ASR Drop, PDD Spike, SQL Queue, Memory High, Intl hourly volume, Intl daily volume
- · P3 (priority 3, information_source): CPS Spike, Packet Loss, anything not matched above
- · Sensor scope defaults to 5+6 (production SBC edge — primary + hot standby)
- · Action-hint blocks are inlined into the email/Slack/Teams renders for known alert classes (Memory High, CPS Spike, etc.) so on-call has triage prompts; ntfy keeps the body terse
VoIPMonitor's native alert engine can't express 'percentage of calls with total duration 0–5s over a 30-minute window'. This script implements alert F1 from leading-indicator-alerts.md: every 5 minutes via cron, query the last 30 min on sensors 5+6, compute total/short counts, and if (volume ≥ 500 calls AND short-call ratio > 40%) fire a synthetic alert by calling voipmon-alert with alert_id 9101 — that hands routing back to the dispatcher so the alert flows through the same channels (and the same 30-min dedup) as every other rule. Has its own 60-min local cooldown on top of dispatcher dedup so a sustained-bad ratio pages once, not every window. Baseline reference: 14-day avg short-call ratio is ~31%, with an Apr 16 fraud-incident peak of 82%.
- · Volume gate: ≥ 500 calls in the 30-min window on sensors 5+6 (skip below this — ratio is meaningless on small samples)
- · Threshold: short-call (0–5s duration) ratio > 40%
- · Pre-fire gate: local cooldown of 60 min (state at /var/lib/voipmon-alert/short-call-scan-state.json)
How priorities are assigned
Mapped from rule severity in code: P1 → 5 (urgent + bypass DND), P2 → 4 (high), P3 → 3 (default). Anything not mapped falls through to 3.
Example payloads
What an alert on this topic actually looks like on a phone.
3 call(s) · sensors 5,6
• 7165551234 → 12125550987 4m32s [US]
+1-DEMO-CUSTOMER · Acme Co
• 7165551234 → 12125550987 3m12s [US]
+1-DEMO-CUSTOMER · Acme Co
+1 moreSensors 5,6
Last 5m: 142 calls · ASR 38% · avg PDD 0.4s · max PDD 1.1s
24h ago: 167 calls · ASR 71% · avg PDD 0.3sNotes
- Setting env var VMON_ALERT_TEST=1 redirects the publish to vaspian-test and prefixes the title with [TEST] — use this to dry-run a new rule before pointing it at the real topic.
- Dedup state lives at /var/lib/voipmon-alert/state.json. Wiping it forces every active rule to re-page on the next run.
vaspian-alerts on server ntfy.sh.