That is the most common way small teams break their own agents. Here is a lightweight process that catches most of it without writing code.

1. Read real runs before you write any tests

Open the last 30 conversations or task runs of one agent and write one line per failure: what went wrong, in plain words. Don't categorise yet. After 30 runs you'll see the same handful of problems again and again: missing information it should have asked for, a tool called with the wrong arguments, an answer that sounds confident but isn't in your docs, a tone problem.

Those clusters are your test categories. They come from your agent, not from a generic checklist.

2. Turn each failure into a test case

A test case is just four columns in a sheet: the input, what a good result must include, what it must never do, and the category. Aim for 20 to 40 cases to start, mostly from real failures plus a few happy paths and a few nasty edge cases (empty input, conflicting instructions, a tool that times out).

3. Use simple checks first, AI judges second

Anything you can check with a rule, check with a rule: did it call the right tool, include the order number, stay under the length limit. For fuzzy things like "is this grounded in our help docs", use an AI judge prompt that answers only PASS or FAIL with a one-line reason.

Then calibrate the judge: grade 20 outputs yourself, run the judge on the same 20, and compare. If it misses real failures, fix the judge prompt before trusting it.

4. Compare every change against a baseline

Run the full test set on the current version and save the results. Before any prompt, model, tool or knowledge change, run it again on the new version and compare category by category. The total pass rate can go up while one category quietly drops. That category is the bug your customer would have found.

A simple release rule works well: block the change if any critical case fails, or if any category drops by more than a set amount. Otherwise ship.

5. Keep the loop going weekly

Every week, read a fresh batch of real runs, add new failures as test cases, and re-run. The test set grows with your agent, and regressions stop reaching customers.

Tools to start with

If you want something running today, the free AI Agent Watchdog is a copy-paste prompt that checks what your agents actually produced, plus a two-minute daily checklist, so you catch an agent that quietly stopped within a day. For the full setup, with 12 failure runbooks, agent contracts and approval gates, see the Self-Healing Agent Team Kit.

Evals reduce risk; they don't guarantee an agent is correct. Keep a human approval step for anything you can't undo.

Nothing turned red. No alert fired. The scheduler says every run "succeeded". And yet the Monday research pack is half a page of filler, and the social drafts are still promoting a campaign that ended two weeks ago.

This is the normal way AI agents fail. Not with a crash, but with a clean exit and a useless output. This article breaks down why that happens, how to recognise each kind of silent failure, and the minimum setup that catches them within a day instead of a week.

Agents rarely fail because the model got worse

When an agent's output degrades, the first instinct is to blame the model or rewrite the prompt. Usually neither is the cause. Agents fail because the world around them moved: a token expired, a spreadsheet changed, a routine was paused and forgotten.

Almost every failure lands in one of five classes. Knowing which class you are looking at tells you where to look and who has to fix it.

1. Access failures

A login expired. An OAuth token was revoked. Someone changed a password. An API key hit its spending cap. A connector needs re-authorisation after the provider changed its policy.

Signature: the run starts, then errors on the first real action. You see a 401, a 403, a "please reconnect your account" message, or, worse, an empty result where there should be data.

Access failures are the loudest of the five, which makes them the easiest. The trap is the "empty result" variant: some connectors return nothing instead of an error when they lose access, and the agent cheerfully reports "no new items today".

2. Input failures

The thing the agent reads changed shape. A column was renamed from Company to Company name. A folder moved. A website the agent scrapes was redesigned. An inbox filter started routing mail somewhere else. A source feed simply went quiet.

Signature: the run "succeeds", but the output is thin, empty, or about the wrong thing.

This is the most dangerous class, because language models are very good at producing plausible text from bad input. An agent that can't find the column it expects will often improvise rather than stop.

3. Capacity failures

Rate limits, quotas, credits, a full disk, a timeout, or a context window that overflowed because a memory file has been growing for three months.

Signature: intermittent. Works on Monday, fails on Thursday, works again on Friday. Failures cluster at the same time of day, often because several agents hit the same provider in the same minute.

4. Instruction drift

The prompt has been edited by three people (or by the agent itself) and now contradicts itself. The brief says "post daily", the brand guide says "never post on weekends". A memory file contains a stale fact that quietly overrides the instructions.

Signature: quality declines slowly. Tone wanders. The agent alternates between two behaviours, or starts asking questions it used to know the answer to.

5. Coordination failures

Two agents write to the same file. A downstream agent depends on an upstream one that stopped. A routine was paused for a holiday and never resumed. Nobody owns the agent anymore.

Signature: gaps. A missing weekly report. A research pack built on last month's data.

The core problem: you are monitoring the wrong thing

Look at the signatures again. Only access failures reliably produce an error. Input failures, instruction drift, and coordination failures produce no error messages at all. Capacity failures produce errors only some of the time.

If your only alerting is "the run crashed", you will miss most of your real failures.

Traditional software monitoring checks exit codes, because in traditional software a wrong result usually comes from a bug that also throws. Agents break that assumption. The worst agent failures exit cleanly with a confident, well-formatted, wrong answer.

So the fix is a change in what you check:

Check outputs, not exit codes.

That means for every agent you need an answer to a question most setups never write down: what does a good output look like, concretely enough that something other than you can check it?

A minimal setup that catches silent failures

You don't need a monitoring platform. You need three plain-text things, and you can set them up in an afternoon.

Step 1: Write down what "done" means for each agent

For every agent, write three to five checkable criteria. "Checkable" means another agent could verify each one by reading the output, without judgement calls.

Bad: "Produces a good research summary."

Good:

Done means:
1. A file exists at /research/output/YYYY-MM-DD.md dated today.
2. It contains at least 5 items.
3. Every item has a source URL.
4. No item is older than 7 days.
5. Total length is between 600 and 1,500 words.

Add an expected output size range. Size is a crude signal, but it is an extremely effective one. A research pack that is normally 1,000 words and suddenly comes in at 180 is almost always a broken input, not a slow news week.

If you can't write "done means" for an agent, that's a finding in itself: the agent can't hit a target nobody defined, and nothing can check it.

Step 2: Make every agent write one line per run

Create a single run log that every agent appends to at the end of every run. Same columns for all of them:

timestamp_iso,agent_name,routine,status,output_path,output_size,notes
2026-10-19T08:02:11+02:00,research,weekly-pack,ok,/research/output/2026-10-19.md,1043,
2026-10-19T08:15:40+02:00,social,drafts,partial,/social/drafts/2026-10-19.md,212,"source sheet missing column 'Campaign'"

Then add this to each agent's instructions:

At the end of every run, append one line to {RUN_LOG_PATH}:
timestamp_iso,agent_name,routine,status(ok|partial|failed),output_path,output_size,notes

"Done" means: {DONE_MEANS}. If you cannot meet all of them, set
status=partial and say which criterion failed in notes.

If an input is missing, empty, or has a different shape than described
here, do not guess. Set status=failed and describe what you saw in notes.

That last paragraph matters more than it looks. It turns the most dangerous failure class (input changed shape, agent improvises) into a visible one. You are explicitly giving the agent permission to fail loudly instead of producing filler.

The run log turns "I think the research bot stopped" into "the research agent's last successful run was 6 days ago and its last three outputs were under 200 words". That is a sentence you can act on.

Step 3: Check the log daily, with something other than your eyes

Once the log exists, checking it is mechanical. For each agent, compare the log against its definition of done:

• Overdue? Is the last successful run older than the agent's cadence? An agent with no log line for twice its cadence is almost certainly paused, disabled, or broken.
• Failing? What were the statuses of the last three runs?
• Missing? Does the latest output actually exist at the path the agent claims?
• Thin? Is the latest output size inside the expected range?
• Wrong? Does it meet each "done means" criterion?
• Trending down? Is output size falling across the last five runs, even if each run individually passes? A 30%+ drop over five runs is worth a look.

You can do this by hand in a few minutes with a spreadsheet filter. Better: hand it to a small, scheduled, read-only agent whose only job is this check. That's usually called a watchdog, and it deserves its own article (it gets one in this series).

Diagnosing what you find

When a check flags an agent, classify the cause before fixing anything. The class tells you who can fix it:

Notice how few of those an agent should fix by itself. A useful rule of thumb: fix automatically only if the fix is reversible, stays inside the agent's own workspace, and doesn't need anyone's permission. One retry after a timeout qualifies. Re-authorising an account, editing instructions, or rescheduling a routine does not.

Three habits that prevent the next one

Catching silent failures quickly is half the job. These three habits stop the most common ones from recurring:

1. Date every fact in agent memory. "As of 2026-10: pricing page is /pricing" ages honestly. "Pricing page is /pricing" lies forever. Give memory files a size budget, too; a notes file that has grown for three months is a common source of both drift and context overflow.
2. Write the resume date when you pause anything. Holiday pauses are one of the easiest ways to lose an agent for weeks.
3. Give every agent a single owner and a kill rule. For example: "pause if no output has been used by a human in 14 days". Without one, dead agents keep running and keep costing money.

The 60-minute version

If you only have an hour today:

1. Pick your most important agent and write its "done means" and expected output size. (15 minutes)
2. Add the run-log instruction above to its prompt. (15 minutes)
3. Tomorrow morning, check the log against the criteria instead of reading the output. (30 minutes the first time, a couple of minutes after that)

That alone moves you from finding out about failures when a post doesn't go out, to finding out the next morning.

Next in this series: a copy-paste watchdog prompt that runs this check for you every morning.