Alert Lookup Index

How alerts reach you

Ghost Mode alerter ─┐
Asset monitor      ─┤
KPI adapter        ─┴──► alerts.sanmarcsoft.com (self-hosted ntfy)
                          │
                          ├─► universal-exports  (operator topic: EVERY asset + EVERY KPI)
                          ├─► ghostmode-alerts   (Phenom client topic: Phenom assets + Phenom KPIs)
                          ├─► kpi-alerts         (scoped KPI feed: all KPI rules)
                          └─► phenom-*           (per-asset team topics, mirrored to ghostmode-alerts)
                          │
                   upstream-base-url → ntfy.sh APNS → iOS (instant, audible)

Topic model (M directive, 2026-06-08; multi-tenant split):

• universal-exports is the operator topic: every alert from every source and every KPI rule lands here. One subscription covers the whole estate.
• ghostmode-alerts is the Phenom client topic: Phenom asset alerts and Phenom-scoped KPI alerts only. SanMarcSoft company KPIs (MRR, churn, signups, Sightengine cost, CI pass rate, bridge agents, exporters) are operator-only and never appear here. Routing is by the alert’s org label (phenom mirrors here; sanmarcsoft does not).
• kpi-alerts is the scoped KPI feed: all Prometheus KPI rules, regardless of owner.
• Per-asset phenom-* topics (phenom-www, phenom-nest, phenom-db-prod, …) fire for the team; all are mirrored to ghostmode-alerts.
• iOS delivery requires upstream-base-url: "https://ntfy.sh" in the ntfy server config (fixed 2026-06-04). Only a topic hash and message id transit ntfy.sh; content stays on our server.

Priority tiers

Master lookup table

Search this table for the title text on the notification.

Prometheus rules (Grafana-visible, see caveat): EndpointDown, SlowResponse, HttpStatusError, SslCertExpiringSoon, SslCertExpiryCritical, SslCertExpired, BlackboxExporterDown, SanMarcSoftExporterDown, MrrDropped, HighChurnRate, CriticalChurnRate, NoSignups48h, SightengineQuotaWarning, SightengineQuotaCritical, AuthBruteForce, LowCIPassRate, CommsContentStale, CommsGenerationFailed, BridgeAgentDown, BridgeAgentHeartbeatFailing, BridgeHealthEndpointUnreachable.

Ghost Mode security alerts (topic: ghostmode-alerts)

Source: osint_surveillance_detector_repo/ghostmode/alerter.py. Tapping any alert opens the gated ops dashboard at https://nest-ops.thephenom.app/ops/ (the Click target is pinned there deliberately - never to an attacker-controlled host).

<host>: CANARY CREDENTIAL from <ip> - P5 urgent

Trigger: an actor submitted a username/password to a honeypot (OpenCanary) service.

Meaning: this is past scanning - someone is actively attempting access and believes the canary is real. Treat as an active, targeted attack.

Respond:

1. Open the ops dashboard (tap the alert) and identify the source IP, country, ASN, and which canary service was hit.
2. Block the IP (and ASN if it recurs) at Cloudflare WAF.
3. Check whether the submitted credentials resemble any real ones; if an actual credential pattern appears, rotate it everywhere immediately.
4. Review surrounding events from the same IP for lateral probing of real services.
5. Preserve the OpenCanary logs for the incident record.

MULTI-DOMAIN recon: <ip> (N sites) - P5 urgent

Trigger: one source IP generated high-severity events on two or more owned domains (thephenom.app, sanmarcsoft.com, verifieddit.com, trusteddit.com) in the correlation window.

Meaning: not random internet noise - this actor knows the portfolio and is mapping it deliberately.

Respond:

1. Open the ops dashboard; note IP, ASN, country, event count, and which domains.
2. Block the IP at Cloudflare across all zones (the alert means per-domain blocking is insufficient).
3. Check the canaries for interaction from the same source (recon often precedes credential attempts).
4. If the ASN is a hosting provider and probing recurs from neighbours, consider an ASN-level WAF rule.

<host>: N recon/blocked hits from <ip> - P4 high

Trigger: a burst of high-severity recon or WAF-block events from a single IP, aggregated into one alert (per-IP aggregation defeats path-walk flooding).

Meaning: scanner or attacker probing one property. The WAF is already doing its job for blocked actions.

Respond:

1. Usually nothing - the aggregation exists so you can glance and move on.
2. If the same IP re-alerts after the 5-minute cooldown, block it at Cloudflare.
3. If paths listed include anything that actually exists (real admin endpoints), check the access logs for hits that were NOT blocked.

<host>: RECON <path> / <host>: BLOCKED - P4 high

Legacy single-event variants of the above (emitted via the MCP tool path). Same response as the aggregated form.

Alert volume capped - P3 digest

Trigger: more than 20 alert-worthy sources in one scan, or the 30-alerts-per-minute global ceiling was hit.

Meaning: alert flood suppression engaged. You are NOT seeing the full picture on the phone.

Respond: open the ops dashboard for the complete event list. A capped scan during quiet hours often means a distributed scan or an attack wave - check the cross-domain correlation panel.

Ghost Mode: all clear - P2 ops heartbeat

Trigger: periodic; confirms the alerting pipeline itself is alive.

Meaning: silence is trustworthy only while these arrive. A missing heartbeat is itself the alert.

Respond (only when it STOPS arriving):

1. Check the ghostmode container: ssh matt@a1.matthewstevens.org "docker ps | grep ghostmode" and its logs.
2. Check ntfy server health: curl https://alerts.sanmarcsoft.com/v1/health.
3. If ntfy is healthy but quiet, the detector stack is down; restart the ghostmode container.

Asset monitor alerts (topics: phenom-*, mirrored to ghostmode-alerts)

Source: ghostmode/asset_monitor.py. Probes every 60s; pages after 2 consecutive failures (debounce); re-pages at most every 30 minutes while still down. Tapping the alert opens the affected service (or the AWS console for RDS/SES).

<Asset>: DOWN - P5 urgent

Trigger: the asset failed its health definition for 2+ consecutive probes (~120s).

Monitored assets and their topics:

* RDS healthy set: AVAILABLE, BACKING-UP, MAINTENANCE, MODIFYING, CONFIGURING-ENHANCED-MONITORING, STORAGE-OPTIMIZATION, UPGRADING. Anything else (stopped, failed, storage-full, …) pages as DOWN.

Respond:

1. Tap the alert - it opens the affected service or console directly.
2. HTTP assets: check the container on the NAS (docker ps, docker logs <name>) and the Cloudflare tunnel for that hostname.
3. RDS: the console link shows the instance state; storage-full and failed states need immediate action, MODIFYING/BACKING-UP never page.
4. SES: check the account-level sending status and any AWS notifications about the account.
5. Expect a re-page every 30 minutes until resolved; <Asset>: RECOVERED (P3) closes the incident.

<Asset>: RECOVERED - P3 digest

The matching DOWN condition cleared. No action; confirm the recovery was yours and not coincidence if you were mid-fix.

Prometheus rules (ops-monitoring)

Source: ops-monitoring/alerting/rules.yml, evaluated by ops-prometheus, visible in Grafana (ops-grafana).

What reaches the Phenom topic

The company KPIs above are SanMarcSoft-internal (revenue, churn, signups, cost, CI, monitoring self-health, voice-bridge infrastructure). They are pinned org: sanmarcsoft in rules.yml, so they never reach the Phenom topic. The endpoint and SSL rules are shared across both tenants and self-classify per probe target, so a Phenom asset outage still reaches you while a SanMarcSoft asset outage does not.

Endpoint group

SSL group

Monitoring self-health

Business group

Cost group

Security group

Deployment group

Comms content group

Bridge agent group (claude-peers voice infrastructure)

Quick reference commands

# ntfy server health
curl https://alerts.sanmarcsoft.com/v1/health

# Voice/bridge health (includes TTS queue + agent registry)
curl http://10.0.0.96:7900/health

# What is running on the NAS
ssh matt@a1.matthewstevens.org "docker ps"

# Ghost Mode ops dashboard (every security alert clicks through to here)
open https://nest-ops.thephenom.app/ops/

# Grafana (Prometheus alerts live here until Alertmanager is wired)
open https://ops.sanmarcsoft.com

Known gaps (as of 2026-06-08)

1. Prometheus rules do not push Resolved 2026-06-08. Alertmanager → ops-kpi-ntfy-adapter → ntfy now delivers all 21 rules, routed by org label (see “These rules now push” above).
2. Heartbeat dependence - if Ghost Mode: all clear stops, nothing else will tell you the security pipeline is dead. Check it when you have not heard it in a while.
3. Manual test alerts use the pattern tk_ tokens minted ad hoc (ntfy token add --expires=1h); revoke after use.
4. Company-KPI classification is label-driven. A SanMarcSoft KPI rule that ships without an org label falls back to a heuristic and could, in principle, mis-route. All current company rules are pinned org: sanmarcsoft; any new one must be too.