What it takes to put an AI agent into a healthcare workflow

The model is the easy part. Consent, audit trails, fallbacks and the people in the loop decide whether an AI agent is safe to run in a healthcare workflow.

By Rajesh, 22 October 2026

A demo of an AI agent booking a patient's follow-up visit takes an afternoon to build. Putting the same agent in front of real patients takes months, and almost none of that time goes into the model.

The gap is everything around it. A healthcare workflow carries obligations a demo never meets: whose data the agent can see, what it is allowed to do on its own, what happens when it is wrong, and who answers for the outcome. Here is how I work through those questions before an agent goes anywhere near production.

Start with the workflow, not the model

Pick one workflow and write it down step by step as it runs today, including the workarounds. Rescheduling a missed appointment, for example, touches the patient's preferences, the clinician's calendar, visit types, referral rules, insurance checks and the reminder system. Most of the difficulty lives in those edges, not in the conversation.

Then mark each step: which ones need judgement, which are lookups, which change a record. That map tells you where an agent can help and where it must not act alone.

Decide what the agent may do on its own

I find it useful to give every action one of four levels:

1. Suggest. The agent proposes; a person decides and acts.
2. Draft. The agent prepares the action, such as a message or a booking, and a person approves it.
3. Act with confirmation. The agent acts after the patient or staff member explicitly confirms.
4. Act. The agent acts and reports what it did.

Start everything one level lower than feels necessary. Move an action up only when the evidence says it is safe, and write down who made that call. Anything clinical, such as triage advice or a medication question, stays at the first level and goes to a clinician.

Treat consent and data access as design inputs

An agent should see the least data it needs for the step it is on, not the whole chart because the API happens to return it. Decide up front:

• what the patient has agreed to, and how the agent checks that before it acts
• which fields the agent may read and write, enforced by the integration layer, not by the prompt
• where conversation data is stored, for how long, and whether any of it leaves your environment
• whether your model provider's terms allow this use with health data, and whether the right agreements are in place

Instructions in a prompt are not access control. If the agent must never see a field, it should never receive it.

Keep an audit trail you could show a regulator

For every action, record what the agent saw, what it was asked, which tools it called with which inputs, what it returned, and who approved it. Store the model and prompt versions alongside, because "the agent did this" means little if you cannot say which agent.

The test is simple: six months from now, could you explain to a patient, a compliance officer or a court exactly why a specific booking was moved? If not, the trail is incomplete.

Design the fallbacks before the happy path

Agents fail in ways ordinary software does not. They misunderstand, they sound confident when they are wrong, and they occasionally do something no one predicted. Plan for it:

• Hand-off rules. Name the situations where the agent stops and passes the conversation to a person: distress, anything clinical, repeated misunderstanding, a request outside its scope.
• Safe defaults. When a call to the scheduling system fails or times out, the agent says so plainly and does nothing it cannot confirm.
• A way out. A patient can always reach a person, and the agent tells them how.
• A kill switch. Operations staff can turn the agent off, or down a level, without a release.

Test it like a clinical tool, not a chatbot

Before launch, build a set of test conversations from real cases, de-identified, including the awkward ones: the patient with two accounts, the caregiver booking for a parent, the request in a second language. Run every change to the prompt, model or tools against that set, and block the release when results get worse.

After launch, keep measuring. Track completed tasks, hand-offs, corrections staff had to make and complaints. Read a sample of conversations every week. Agents drift as the systems and people around them change.

The people in the loop need a real job

"Human in the loop" fails when the human is an afterthought: a queue nobody owns, a screen that hides the context, an approval button pressed out of habit. Give reviewers the agent's reasoning and the data it used, keep their queue small enough to read properly, and make it clear who is accountable when they approve.

When it works, the agent handles the routine back-and-forth, and people spend their time on the cases that need them. That is the point of the whole exercise.

A short checklist

• One workflow, mapped as it runs today
• An autonomy level for every action, with a named owner for each change
• Data access enforced in the integration, not the prompt
• A complete, versioned audit trail
• Hand-off rules, safe defaults and a kill switch
• A test set from real cases that every change must pass
• Reviewers with the context, time and authority to do the job

If you're working through this for a non-profit or community health organization, I'm glad to help. You'll find my email at the bottom of the home page.