A dead man's switch for every AI job we run

The failure mode of an automated business isn’t the job that crashes. It’s the job that stops running and nobody notices for three weeks. A scraper’s session expires. A cloud automation hits its execution quota. A Managed Agent’s schedule gets paused to save money and never unpaused. Every one of those looks identical from the outside: silence.

By September I had about thirty of these things at Every Man Jack. n8n workflows on a cloud instance and a self-hosted one, cron jobs on three DigitalOcean droplets, Vercel crons across four apps, and a couple of Anthropic Managed Agents on Console schedules. My first attempt at watching them was a hand-rolled collector inside our internal AI Command Center: a five-minute cron that polled every runtime’s API, wrote rows to Postgres, and drew tiles. It worked, and it was the wrong shape. Nothing watched the watcher, every new runtime meant a new provider module, and I was maintaining a small monitoring product instead of running a business.

Two problems, two tools

The realization that made this cheap: “did it run” and “what happened” are different problems, and the expensive tools bundle them together.

“Did it run” is a dead man’s switch. Every job gets a URL. It hits the URL when it finishes. If the URL isn’t hit on schedule, someone gets paged. That’s the entire product, and Healthchecks.io does it for $20 a month. Each check knows its cron expression and a grace period, so “the nightly pull didn’t happen” fires at 1:00 AM, not when a human opens a dashboard.

“What happened” is a log. One JSON event per run with a shared shape: agent_id, run_id, event (started, finished, failed), duration_ms, error. That goes to Axiom, whose free tier is absurd for this volume, and whose query language lets me ask “which agents failed more than twice this week” in one line.

Datadog would have done both. It would also have cost more per month than the entire agent fleet costs to run, for a company with one person on the infrastructure.

One script on the droplets

The droplet jobs all end the same way now. The cron line does the work, then calls one helper with the job’s key and an ok or fail:

1
0 13 * * * root cd /opt/confido-scraper && (node src/run.js >> /var/log/confido-scraper.log 2>&1 \
2
  && aioc-heartbeat hb.confido-scraper ok /var/log/confido-scraper.log \
3
  || aioc-heartbeat hb.confido-scraper fail /var/log/confido-scraper.log)

The helper looks up the key’s ping URL in an env file, appends /fail when the run failed, and posts the tail of the log as the ping body so the failure text shows up in Healthchecks without anyone SSHing in. Then it sends the same run to Axiom as one event:

1
EVT=$([ "$STATUS" = "fail" ] && echo failed || echo finished)
2
ERR=""; [ "$STATUS" = "fail" ] && ERR=$(printf "%s" "$LOG" | tail -c 800)
3
AX=$(jq -nc --arg a "$HC_KEY" --arg e "$EVT" --arg h "$(hostname)" --arg err "$ERR" \
4
      --argjson d "${DISK:-0}" --argjson m "${MEM:-0}" \
5
      '[{agent_id:$a, event:$e, runtime:"droplet", host:$h, disk_pct:$d, mem_pct:$m,
6
         run_id:(now|tostring), _time:(now|todate)} + (if $err == "" then {} else {error:$err} end)]')
7
curl -s -m 15 -X POST -H "Authorization: Bearer ${AXIOM_TOKEN}" -H "Content-Type: application/json" \
8
  -d "$AX" "${AXIOM_INGEST_URL}"

n8n got the same treatment with less code. Each scheduled workflow ends in an HTTP node that pings its check, and each instance has one shared error workflow that posts the failing workflow’s name and error to /fail. Vercel crons call a ten-line helper. The Managed Agents were the awkward case: the Console has no end-of-run hook, so a small cron reads each deployment’s last_run_at after its window and pings on the agent’s behalf.

The registry is the product

The status page itself is now a view, not a system. There’s no collector and no database. A typed registry in the app lists every job with the business process it belongs to, the runtime it lives on, and its Healthchecks slug and Axiom agent id:

1
{ key: "confido-scraper", healthcheckSlug: "confido-scraper", axiomAgentId: "hb.confido-scraper",
2
  process: "finance", kind: "scheduled", runtime: "droplet", owner: JM,
3
  name: "Confido nightly pull",
4
  sentence: "Pulls the trade-spend ledger into the warehouse every night." },

On request the page reads Healthchecks and Axiom live, joins both to the registry, and applies a handful of pure rules: a check that’s up is green, in grace is amber, down is red; an agent whose last event was failed is red; three failures in a day is red, one or two is amber; the worst colored status wins for the process card. Sixty-second memo, no cron, no tables. The registry is also the catalog, so the page doubles as the answer to “what AI do we actually run” for people who will never open Healthchecks.

Alerting is boring on purpose. Healthchecks posts to a Slack channel when a check goes down. Axiom has one match monitor:

1
['emj-agent-runs']
2
| where event == "failed" and not(agent_id startswith "ops.")
3
| extend error_text = column_ifexists("error", "")
4
| project-keep _time, agent_id, run_id, error_text

That column_ifexists is not decoration. Axiom rejects a query that names a field no event has ever carried, and a monitor definition that 400s on save is a fun way to lose twenty minutes.

What I’d tell another CPG shop

You don’t need an observability platform to know whether your automations are alive. You need every job to phone home when it finishes and something dumb and reliable to notice when it doesn’t. Separate that from the “why did it fail” question and both halves get cheap.

The part I’m proudest of is what got deleted. The collector, its provider modules, the heartbeat route, four Postgres tables and seven API keys the old design needed to poll everything read-only. The new page reads two vendors with two read-only tokens. Fewer secrets, fewer moving parts, and the one thing that can break is watched by the one thing that can’t.