Keep your monitors. Fix your alerts.

• One flapping service, forty messages.DOWN, UP, DOWN every minute, and every one of them buzzes your phone.
• The failed backup looks like the other twenty-nine.The nightly report arrives by mail. Nobody opens the one that matters.
• The job that stopped in March.Nothing told you it failed, because nothing ran.
• Everyone saw it. Nobody is on it.The alert went to the group chat, and there is no way to say “mine”.

1. Sources

   The apps and hosts you want to hear about, and the tools that watch them.

   Proxmox, Uptime Kuma, Alertmanager, Grafana, Beszel, cron, curl

2. Events

   Every alert becomes the same shape: what happened, to what, and how bad.

   Title, severity, resource, a key that says “this is the same problem”

3. Rules and incidents

   Repeats fold into one incident. Recoveries close it. Your rules decide who hears about it, and when.

   Routing by severity and source, quiet hours, silences

4. Destinations

   Where you want to be told. Acknowledge in one place, and the rest stop repeating it.

   Telegram, Slack, Discord, webhooks, the iPhone app

• Repeats become one incident

  Each new occurrence is counted on the open incident instead of sent to you again. A service that flaps is marked as flapping, and a failure that comes back soon after it recovered reopens the same incident.

  Critical  nextcloud is down            14 events, flapping

• Acknowledge it once

  Acknowledge from the dashboard or the iPhone app, and re-notifying stops. Telegram and Discord messages are edited in place to show who has it; channels that can’t edit a message hear about the recovery.

  Acknowledged by you, 03:17   re-notify stopped

• Recovery closes the incident

  An “up” from Uptime Kuma, a resolved alert from Alertmanager or a success event with the same key resolves the incident, and the incident records how long it lasted.

  Resolved  nextcloud is down        down for 14 min

• Rules run on the server, for every channel

  Route by severity and source, hold the non-urgent until morning with quiet hours, and silence a host during maintenance. It happens before anything is sent, so every channel follows the same rules.

• Heartbeats for jobs that fail by not running

  Give a backup script or a cron job a ping URL. If it doesn’t check in on time, or reports a non-zero exit, you get an incident with its log attached.

• Proxmox, guided

  Proxmox VE and Proxmox Backup Server send to InfraInbox through their built-in webhook target. The setup wizard writes the configuration for you, and is designed to take about five minutes.

How each one connects

# Later, the same dedupKey with "severity": "success" resolves it
curl -fsS https://infrainbox.example.com/v1/events \
  -H "Authorization: Bearer $INFRAINBOX_KEY" \
  -H "Content-Type: application/json" \
  -d '{"title": "Backup failed on nas-01",
       "severity": "error",
       "dedupKey": "backup:nas-01"}'

# crontab: report the exit status, and the log if it failed
0 3 * * * /usr/local/bin/backup.sh > /tmp/backup.log 2>&1; \
  curl -fsS --data-binary @/tmp/backup.log \
  https://infrainbox.example.com/v1/heartbeats/$HB_KEY/$?

More recipes in the cookbook

• End-to-end encrypted push.Alerts reach the app through our relay as ciphertext. The relay can’t read them, self-hosted included.
• No telemetry.The self-hosted server doesn’t phone home. Events, incidents and settings stay on it.
• Secrets stay secret.Destination tokens are encrypted at rest and never written to logs or error messages.
• It never touches your infrastructure.InfraInbox receives and routes alerts. It has no shell, start or stop actions.

• Not a monitor.It doesn’t probe your services. It listens to the tools that do.
• Not a push pipe.ntfy, Gotify and Pushover do that well. InfraInbox decides what is worth sending.
• Not an on-call pager.v0.1 has no rotas or escalation chains.
• Not a Telegram bot.Telegram is one destination among several.

All questions