vaspian-telescope

The FastAPI process publishes two event classes through the shared ntfy helper (integrations/ntfy.py — a best-effort async wrapper that logs and swallows publish failures so a dead ntfy.sh never blocks a test). (1) Test submitted: one push per fired batch when a voice test is enqueued, deduped in Redis with SET NX (key ntfy:batch_submitted:<batch_id>, TTL 1h) so multi-CID submissions that share a batch fire once. (2) Phone connectivity: a background watcher polls the ADB handset snapshot every ≥15s and fires only on edges — connected→disconnected (offline) and disconnected→connected (back online). The first-ever observation seeds state silently; steady state stays quiet.

• · Test submitted — voice batch enqueued (priority low, tags arrow_forward + telephone_receiver; Redis NX dedup, TTL 1h)
• · Phone offline — handset drops off ADB, connected→disconnected edge (priority high, tags warning + iphone)
• · Phone back online — handset reconnects, disconnected→connected edge (priority low, tags white_check_mark + iphone)

Each per-carrier voice worker publishes two event classes. (1) Batch complete: once every test row in a batch reaches a terminal state, the worker aggregates clean / labeled / undelivered counts and a pass %, then fires once (Redis publish_once, key ntfy:batch_completed:<batch_id>, TTL 24h) — the same event the auto-report email rides on. Tone keys off the pass rate. (2) Worker error: on an unhandled job exception, fired once per (carrier, exception-class) with a 300s cooldown (key ntfy:worker_err:<carrier>:<ExcClass>) so a stuck dependency failing 50 calls in a row pages once, not 50×. publish_once drops the dedup key if the publish itself raises, so a transient ntfy.sh 5xx doesn't silently swallow the alert.

• · Test complete, pass ≥ 80% — priority low, tags white_check_mark + telephone_receiver
• · Test complete, pass 50–79% — priority default, tags warning + telephone_receiver
• · Test complete, pass < 50% — priority high, tags x + telephone_receiver
• · Worker error — unhandled job exception (priority high, tags rotating_light + robot; 300s cooldown per carrier + exception class)

A shell health monitor that runs from a systemd timer every 5 minutes on the prod host and catches the things that quietly degrade between deploys: root-disk usage, memory pressure, the full expected docker container set (21 containers — api, dashboard, nginx, postgres, redis, minio, asterisk, cloudflared, adb-watchdog, plus the per-carrier voice/sms/poller/importer workers), ADB seeing all 3 handsets, and the inbound Cloudflare tunnel (a 200/302 from telescope.vaspian.me). State is written per alert key to /var/lib/telescope/sysmon.state; an alert fires only on the healthy→unhealthy edge and stays silent while the condition persists. Recovery (unhealthy→healthy) fires once at priority 2. Every title is prefixed 'telescope · '.

• · Disk usage critical — root fs ≥ 85% (priority high)
• · Memory pressure — used ≥ 90% (priority high)
• · Container(s) down — any of the 21 expected containers not running (priority max)
• · Phones offline — adb sees < 3 handsets (priority high)
• · Cloudflared tunnel unhealthy — telescope.vaspian.me not 200/302 (priority max)
• · Recovery — any of the above clearing fires once (priority low)