Letting an AI agent touch real work — customer replies, contract redlines, quotes — feels risky because it is. The answer isn’t to avoid agents; it’s to build the controls that make their behaviour measurable and their mistakes cheap. Here’s the checklist we use on every build.

Write the evaluations first

Before we write a single prompt, we build an evaluation set: 100–300 real examples from the client’s history, each with the correct outcome agreed by the people who do the work. It’s the agent’s exam, and it never changes without review.

Every change to a prompt, model or tool is scored against that set before it ships. If accuracy drops in any category, the change doesn’t go out. That one habit prevents most of the regressions we see in other teams’ systems.

Set confidence thresholds

An agent should know when it doesn’t know. We ask it to score its own certainty and combine that with simple checks — missing fields, unusual amounts, unfamiliar senders. Anything below the threshold goes to a person. We start strict and loosen the threshold only when the data shows it’s safe.

Design the human review

Human-in-the-loop only works if reviewing is faster than doing the task from scratch. Show the reviewer what the agent proposes, why, and what it was unsure about. Make approve, edit and reject one click each — and log which one they chose.

The goal isn’t an agent that never makes mistakes. It’s a system where mistakes are caught early, cheaply and visibly.

Those review decisions become new evaluation examples, so the system gets measurably better every week.

Keep an eye on reviewer fatigue, too. If people approve hundreds of suggestions a day without changes, they stop reading carefully. Sample a share of approved cases for a second check, and rotate reviewers on high-volume queues.

A launch checklist

Before any agent goes live, we confirm that:

• An evaluation set exists, and the agent passes the agreed accuracy bar.

• Confidence thresholds and hard rules route uncertain cases to people.

• The agent can only use the tools and data it needs — nothing more.

• Every run is logged with inputs, outputs and a reasoning summary.

• Alerts fire on error spikes, cost spikes and unusual volumes.

• A named owner can pause the agent in one click.

Start in shadow mode: For the first week, run the agent alongside the team without acting on its output, and compare its decisions with theirs. It’s the cheapest way to find edge cases before your customers do.