Designing human-in-the-loop AI that people can trust

Language models can now draft, summarize, classify and recommend well enough to be useful across a wide range of business work. They are not reliable enough to be left alone with decisions that affect someone's money, health, access or legal position. Their mistakes are fluent and confident, and they do not announce themselves.

Human-in-the-loop design keeps the benefit without giving up accountability. The model does the reading, retrieval and drafting; a qualified person makes the decision with the evidence in front of them. Done well, this is faster than a manual process and safer than an automated one. Done badly, it produces a reviewer who approves everything and carries the blame when something goes wrong.

Why autonomy is the wrong default

The right level of automation depends on two things: how much harm an error can cause, and how easily it can be reversed. Letting a model act alone is reasonable when mistakes are cheap, visible and easy to undo, such as tagging support tickets or drafting internal notes. It is the wrong default when a decision is hard to reverse or affects people's rights and livelihoods: credit, insurance claims, eligibility, medical triage, hiring.

Accountability cannot be delegated to a model either. Customers, auditors and regulators expect a person or organization to answer for a decision, and some regulation makes oversight explicit. The EU AI Act, for example, sets human oversight requirements for systems it classifies as high-risk.

Make the level of autonomy a documented decision for each task, and change it based on evidence from evaluation and monitoring.

Define what the human actually does

"Human in the loop" means little until the role is specified. For each decision point, define the available actions:

• Approve: the reviewer asserts that the output is correct and supported. Approval is a positive statement, not the absence of an objection.
• Edit: the reviewer corrects the output, and the changes are captured.
• Reject: the output is discarded with a reason code, and the case continues on a manual path.
• Escalate: the case goes to someone with more authority or expertise, with its full context attached.

Then define who is qualified to review, what they are accountable for, how long a careful review realistically takes, and what happens when the model declines to answer. Reviewers need the time and authority to disagree. If throughput targets assume nearly everything will be approved, the review is decorative, and the reviewer becomes what researcher Madeleine Clare Elish called a "moral crumple zone": the person who absorbs responsibility for failures of a system they could not meaningfully control.

Ground the model in sources people can check

A model answering from its training data alone may be out of date, generic or wrong for your context. Retrieval-augmented generation (RAG) retrieves relevant passages from an approved, current set of documents, such as policies, contracts or case files, and instructs the model to answer from them.

RAG reduces errors; it does not eliminate them. Retrieval can miss the relevant passage, and the model can misread a source or cite one that does not support its claim. Design for that:

• Cite at the level of claims, linking each statement to the exact passage behind it, so a reviewer can verify it in seconds.
• Make abstaining acceptable. When retrieval finds nothing relevant, the system should say so and route the case to a person.
• Apply access control at retrieval time, so users only receive answers grounded in documents they are allowed to read.
• Curate the corpus. Outdated or conflicting documents produce outdated or conflicting answers, so the collection needs an owner.

Design the review screen against automation bias

People tend to over-trust automated recommendations. They accept suggestions they would have questioned had they formed their own judgment first, and they miss problems the system did not flag. Human factors research calls this automation bias, and it tends to strengthen when the system is usually right and reviewers are busy. The interface either counters it or amplifies it:

• Put evidence beside each claim, with the supporting passage highlighted.
• Surface what is missing or uncertain: fields the model could not fill, conflicting sources, business rules that failed. A confidence score the model reports about itself is not a reliable substitute.
• Make disagreement as easy as agreement, with inline editing and one-click rejection with a reason code. Do not preselect "approve" or put it where the eye lands first.
• Require active checks for high-stakes cases, such as confirming key figures, or recording the reviewer's own assessment before showing the model's recommendation.
• Watch for rubber-stamping. Very short review times and near-universal approval are warning signs. Periodically include cases with a known answer to confirm that reviews catch errors.

Evaluate before release and after every change

Before launch, build an evaluation set: representative real cases, anonymized where necessary, each with an expected outcome or grading criteria agreed with domain experts. Include edge cases, adversarial inputs and cases where the right behavior is to abstain or escalate. Agree acceptance criteria with the business before seeing results, including which errors are unacceptable at any rate.

Score with a mix of methods: exact checks for structured fields, rubric-based grading for free text (by people, or by automated graders calibrated against human judgment), and expert review of a sample. Treat prompts, retrieval settings and model versions as code: version them, review changes, and run the evaluation set in CI whenever any of them changes. Pin model versions where your provider allows it, so upgrades happen on your schedule. Reviewer corrections in production are a steady source of new test cases.

Keep an audit trail you can use

For every AI-assisted decision, record enough to reconstruct it later:

{
  "caseId": "case-20931",
  "inputRef": "intake/case-20931/v2",
  "retrieved": ["policy-handbook#4.2@v12", "pricing-rules#7@v5"],
  "model": "provider-model-snapshot-id",
  "promptVersion": "case-summary@14",
  "outputRef": "drafts/case-20931/1",
  "reviewer": "user-2291",
  "decision": "edited",
  "reasonCode": "missing-exclusion",
  "decidedAt": "2026-11-05T14:32:10Z"
}

That record answers disputes and audits, makes debugging possible and feeds the evaluation set. It will often contain personal data, so apply the same access controls, retention limits and data-protection rules as the source system, and keep it append-only.

Monitor quality, drift and cost

Evaluation tells you the system was good enough at release; monitoring tells you whether it still is.

• Quality: edit, rejection, escalation and abstention rates by case type, plus expert review of a sample of approved outputs, because approved is not the same as correct.
• Drift: inputs change as products, formats and regulations change, and providers update models. Track input characteristics and retrieval results, and re-run evaluations on fresh samples.
• Cost and latency: cost per completed case rather than per request, and latency as reviewers experience it. Alert on spikes, which can point to retries, loops or unexpectedly large inputs.

Decide in advance what triggers action: rolling back a prompt or model version, reducing autonomy or pausing the feature.

Secure the system around the model

Prompt injection is the defining risk: instructions hidden in user input or retrieved content, such as a document or email, can override the behavior you intended. The OWASP Top 10 for LLM Applications (opens in a new tab) lists it first and notes that it is unclear whether fool-proof prevention methods exist. So limit what a successful injection can do:

• Treat model output as untrusted: validate it against an expected format with deterministic code, and encode it before display.
• Give the model the least privilege it needs, and enforce authorization in application code, not in prompt instructions.
• Require human approval for actions with external effects, such as sending, paying or deleting. Here the loop is a security control as well as a quality one.

Data leakage needs the same discipline: permission-aware retrieval, no secrets in prompts, and a clear understanding of what your model provider retains. Add injection and leakage attempts to your evaluation set.

Where to start

1. List your AI-assisted decisions, classify each by consequence and reversibility, and set its level of autonomy explicitly.
2. For one consequential workflow, define reviewer actions and reason codes, and redesign the review screen around evidence.
3. Build a first evaluation set with domain experts and run it in CI before the next prompt or model change.
4. Capture complete audit records now, even before you build dashboards on them.
5. Review every tool and data source the model can reach for injection and leakage risk.

Our AI engineering team designs these controls in from the first prototype.