tower.dunn.dev

architecture

How tower works

A prober Worker checks and mails; a page Worker serves the page and hands requests to it.

Two Workers

The prober runs on a cron each minute, runs the checks that are due, writes results to D1, sends alert mail through the send_email binding and answers the page over Workers RPC. The page Worker serves the page, /api/status, /beat/* and /relay/*, and hands what it receives to the prober.

HTTP RPC owns alerts OUTSIDE readers, senders page, /beat, /relay PAGE WORKER no secrets no storage minute cron PROBER WORKER cron checks sole D1 owner D1 history state, outbox EMAIL alert mail send_email HTTP RPC owns alerts OUTSIDE readers, senders page, /beat, /relay PAGE WORKER no secrets no storage minute cron PROBER WORKER cron checks sole D1 owner D1 history state, outbox EMAIL alert mail send_email
The page Worker faces the internet and holds no secrets or storage. The prober owns the data and sends the alert mail; the page Worker mails only when the prober does not answer.

A check's life

A probed check starts up, and each passing run keeps it there. The prober runs on a cron each minute and runs the checks that are due. When runs fail, it stays up and counts the failures until down_after runs in a row have failed.
When down_after runs in a row have failed, the check goes down and the change queues one DOWN mail in the outbox. The failure reason goes to that mail and the Worker log only, never the public page. A maintenance note suppresses alerts for its checks inside its window.
Once runs pass again, the check stays down and counts the passes, until up_after runs in a row have passed. Meanwhile the public page shows it down by name, with how long it has held that status, its uptime and its response times, while its address stays behind.
When up_after runs in a row have passed, the check is up again and the change queues one UP mail, so tower mails once when a check goes down and once when it recovers. Each mail waits in the outbox until it sends: a failed send is tried again on the next run.
fail limit pass limit DOWN UP STATE up runs pass STILL UP failing runs fail until down_after STATE down DOWN mail STILL DOWN passing runs pass until up_after STATE up again UP mail OUTBOX one mail per change retried each run fail limit pass limit DOWN UP STATE up runs pass STILL UP failing runs fail until down_after STATE down DOWN mail STILL DOWN passing runs pass until up_after STATE up again UP mail OUTBOX one mail per change retried each run
A probed check changes state after down_after or up_after runs in a row; a heartbeat goes down at its first missed deadline.

Five check types

http

Requests an address and judges the response.

tcp

Opens a connection to a port. It sends no data.

dns

Resolves a name through 1.1.1.1 over DoH.

tls

Reads a zone's edge certificates through the Cloudflare API.

heartbeat

Waits for a request from your job within its interval plus a grace.

Names only

Failure reasons go to the alert mail and the Worker log only. Heartbeat and relay tokens are stored as sha256 hashes, so the plaintext never enters your config. A check that has not run recently reads as unknown, so a stopped cron cannot leave a stale green page.

PROBER holds it all checks, results CROSSES names, status uptime, times PUBLIC page and API /api/status STAYS BEHIND addresses urls, hosts, ports failure reasons PRIVATE mail and log reasons go here PROBER holds it all checks, results CROSSES names, status uptime, times PUBLIC page and API /api/status STAYS BEHIND addresses urls, hosts, ports failure reasons PRIVATE mail and log reasons go here
Names, status, uptime and response times cross to the public page; addresses and reasons stay behind.

Watching itself

The prober flags any check with no conclusive result for three of its intervals and mails a self-check failure naming it, then an OK when results return. A probe that stays inconclusive or misconfigured, such as a tls check with an unset token, counts as no result, so a check that stops producing results still reaches your inbox.
The page Worker's minute cron asks the prober to watch its last run. If no run has written results for three of the shortest intervals, the mail says probing stalled, then an OK follows once probing resumes. Because the page Worker's cron asks, not the prober's own, the mail survives a dead prober cron.
If the prober does not answer at all, the page Worker mails that probing stalled and the prober did not answer, at the top of each hour until it does. This is the only mail the page Worker sends: it holds no secrets or storage, and the prober sends every other alert.
When both Workers are dead, an account or platform failure, nothing can mail. A daily digest at a time you choose is the alarm: a missing digest means something is wrong. A normal day's digest is the subject all up with no body. An outside poller can also read /api/status, which reports the last run, the oldest unsent mail and stuck checks.
stuck digest watch no reply poll PROBER flags checks no result for 3 intervals EMAIL SELF-CHECK FAILING mail YOU read the digest none is the alarm PAGE WORKER asks watch every minute OUTSIDE poller optional stuck digest watch no reply poll PROBER flags checks no result for 3 intervals EMAIL SELF-CHECK FAILING mail YOU read the digest none is the alarm PAGE WORKER asks watch every minute OUTSIDE poller optional
The prober flags stuck checks, the page Worker flags a stalled prober, and a missing digest covers both being down.

Relays

A relay accepts an incident.io-format webhook at POST /relay/incident, as HyperDX sends, and mails on transitions only. tower reads each alert's deduplication key as its key. A firing for a closed key opens it and mails FIRING. Titles, descriptions and keys reach the alert mail only, never the page or /api/status.
A repeat firing of an open key renews its lease without mail, and the request answers duplicate. A sender that keeps re-notifying therefore holds the key open in silence, so set the relay's grace above the sender's re-notify gap, at most seven days.
A resolved closes the key and mails RESOLVED. A resolved for a key that is already closed is a duplicate and sends nothing, because a relay mails on transitions only. A later firing of the same key opens it again and mails FIRING again.
A key not renewed within its grace closes with STOPPED REPORTING, which also covers a lost resolved. Every transition shares one limit: at most five transitions per relay per minute mail at once; the rest wait for the next run and, when more than one is waiting, arrive as one mail.
fire close no repeat mail mail mail KEY closed no mail KEY OPEN FIRING repeats renew without mail KEY RESOLVED sender closed GRACE LAPSED STOPPED REPORTING MAIL 5 a minute rest batched one ALERTS mail fire close no repeat mail mail mail KEY closed no mail KEY OPEN FIRING repeats renew without mail KEY RESOLVED sender closed GRACE LAPSED STOPPED REPORTING MAIL 5 a minute rest batched one ALERTS mail
A relay key mails when it opens, when it closes and when it goes quiet.

Limits

  • A tcp check against a Cloudflare-proxied address, or one Cloudflare blocks, reads as down.
  • A tcp check sends no data, so a service that accepts and then fails reads as up.
  • A tls check covers edge certificates. An origin certificate failure shows as http 526.
  • Checks run from one Cloudflare location per cron run, so a failure near it can read as down.
  • Relays have no silences or maintenance windows.
  • A relay holds at most 100 open keys. A firing past that is rejected, and one SELF-CHECK FAILING mail says so.
  • If both Workers are dead, nothing mails, and only the missing digest says so.