How to keep a background AI agent from failing silently

A background AI agent silent failure hides in plain sight. A heartbeat, a cron wrapper, and a watchdog make a dead scheduled agent alert you within the hour.

0:00
How to keep a background AI agent from failing silently

Our morning briefing agent ran every day for a week, and then it stopped, and we did not notice for three days. The cron job still fired at 7:00 AM. The Hermes runtime still launched. But a model update had renamed the endpoint the skill called, the run threw an error halfway through, and the agent exited without sending anything. No briefing arrived, and because a missing message looks exactly like a quiet morning, nothing told us the system was broken. It had simply gone silent.

This is the failure mode that matters once you move an AI agent into the background. Setting one up is the easy part: a memory file, a skill, a cron entry, and you have an agent that runs without you. The hard part is the day it breaks, because a background agent that fails produces the same output as a background agent with nothing to report: nothing. Silence is both the success state and the failure state, and telling them apart is the entire discipline of running unattended software.

Why background agents fail silently

A chat agent cannot fail quietly. You are watching it. When it errors, stalls, or hallucinates, you see it happen and you retry. The human in the loop is also the monitoring system.

Move the agent to a schedule and you remove that monitor. Now it runs at 7:00 AM whether or not you are awake, and the only evidence it ran is the result it delivers. When the result stops arriving there is no error on your screen, because you are not looking at a screen. The whole point of an invisible system is that it does not demand your attention, which means it also cannot easily ask for it when something goes wrong.

Local agents add a second layer of fragility. A cloud API is a stable target; your own stack is not. LM Studio can be closed. A model can be swapped for one with a different context limit. A path inside a skill can move. Any of these breaks the run, and none of them announces itself. The agent we lost for three days was a casualty of exactly this: a working system that quietly stopped working because the ground under it shifted.

How to add a heartbeat to every scheduled agent

The fix is to make success loud instead of silent. Every scheduled agent should end by recording that it finished, and something should check for that record.

We do this with a heartbeat file. The last step of every background run writes a single line to a log: the timestamp and the exit status. A run that completes writes 2026-07-11 07:00 ok. A run that dies before that line simply does not write it. The absence of a recent, healthy line is now a signal we can detect, which is the one thing silence can never be.

The pattern is old and well proven outside AI. A dead man’s switch fires precisely because a signal stopped arriving, and services like Healthchecks are built entirely around the same idea: you tell them how often a job should check in, and they alert you when a check-in is late. You do not need the service. You need the concept. A job that must announce its own health turns silence from an ambiguous state into an alarm.

The wrapper does the recording, not the agent. We do not trust a failing agent to log its own failure, because the failure might be the thing that stops it reaching the logging step. Instead the cron entry calls a small wrapper script that runs the agent, captures its exit code, and writes the heartbeat itself. If the agent crashes, the wrapper still runs and still records a non-zero exit. The monitoring lives one level above the thing it monitors.

How to build a watchdog that catches a dead agent

A heartbeat is only useful if something reads it, so we run a second agent whose only job is to watch the first.

The watchdog runs on its own schedule, more often than the agents it checks. Every hour it reads the heartbeat log and asks one question: has each agent that should have reported by now actually reported, and did it report success? If the morning briefing was due at 7:00 and the log shows no healthy line after 7:05, the watchdog sends the alert the briefing agent could not: “Daily briefing did not run this morning. Last healthy run was yesterday at 07:00.”

This inverts the failure. Without a watchdog, a broken agent stays invisible until you happen to notice its output is missing, which for us took three days. With one, a broken agent produces an alert within the hour, and the alert carries the detail you need to fix it: which agent, when it last worked, and the exit code it died with. We keep the watchdog deliberately simple and deliberately separate. It shares no code with the agents it monitors, so the bug that kills one of them cannot also blind the thing watching.

What to do when the agent runs but returns garbage

Not every failure is a crash. The harder ones complete successfully and deliver something wrong.

The most common with a local model is temporal drift. Ask an agent to prepare a briefing for tomorrow and it may reason from the date in its training data rather than the actual system clock, then confidently brief you on the wrong day. Nothing errors. The exit code is zero. The heartbeat is healthy. The output is simply false. We fix this by never letting the model supply the date. The skill’s first step runs date in the terminal, and we instruct the agent to treat that output as the only ground truth for every time calculation that follows. Anchor the reasoning to a tool result and the invented date disappears.

The general rule is that a background agent should verify with a tool anything a human would have caught by looking. A person reading a briefing dated for last Tuesday would frown and recheck. The agent will not, so the check has to be built in: pull the real number, the real date, the real file, and reason from those rather than from the model’s memory of them. This is what local tool calling is for, and it is the difference between an agent that assists and one that quietly misleads.

What running unattended actually asks of you

The shift from a chat assistant to a background system is not really a shift in the AI. It is a shift in what you are responsible for. When you drive the agent by hand, you are its error handler. When you schedule it, you have to build the error handling you used to provide by being present.

That is the work the tutorials skip. Setting up the cron job takes an afternoon. Making it safe to walk away from, so the day it breaks you find out in an hour instead of three days, takes a heartbeat, a wrapper, and a watchdog. We built ours only after we lost three mornings to a system we thought was running and was not. The agents that earn a place in the background are not the ones that work while you watch. They are the ones that tell you the moment they stop.

Share this