TrueWatch
Get Started

Human in the Loop AI Agents: Fixing the Review Bottleneck

Nov 17, 2026

Engineering teams adopted AI to move faster, and by one measure it worked. Bain's 2026 survey of senior technology leaders found that developers complete roughly 21% more tasks with AI, but review time rises approximately 91%. Bain's own conclusion is blunt: human review is now the critical constraint. That finding matters for anyone deploying human in the loop AI agents. If every agent output lands on a person's desk, the person becomes the bottleneck, and adding more agents only lengthens the queue.

This post looks at what the data says, why the review load grows faster than the output, and how to design agent handoffs so the human in the loop checks evidence instead of redoing the work.

AI made engineers faster. Review got slower.

Adoption is no longer the question. Google's 2025 DORA research reported that AI adoption among software development professionals has surged to 90%, and that adoption is now linked to higher software delivery throughput. More code, more changes and more tickets are moving through the system.

What has not scaled is the checking. Bain's Tech and Engineering Survey of 293 senior technology leaders, run in April 2026, puts numbers on it: roughly 21% more tasks completed, against roughly 91% more time spent reviewing. The work arrives faster than people can confirm it is right.

The DORA team frames AI as an amplifier. In its words, AI's primary role is "magnifying an organization's existing strengths and weaknesses". A review process that was already stretched does not get easier when the volume going into it rises. It gets louder.

01_after-AI-made-engineers-faster_review-time-chart.png

Why human in the loop AI agents hit a review ceiling

Human-in-the-loop describes systems where people actively participate in the operation of AI, providing guidance, feedback and oversight. For AI agents that act on production systems, that oversight usually means a person reads what the agent found and decides what happens next. The model is sound. The capacity is the problem.

Five times the agents, one reviewer

Bain's respondents anticipate nearly a fivefold increase in agents per team. If each of those agents hands back a bare answer ("the cause is the database"), the reviewer has to rebuild the reasoning before they can trust it. They reopen the dashboards, rerun the queries and read the logs again. At that point the agent saved very little, and the reviewer now has five times as many answers to verify.

The same Bain article describes how one high-volume engineering organisation handles this: about 1,300 AI-authored pull requests merged per week, all going through the same human review as any other change. Human review stays. What has to change is how much work each review takes.

The same pattern is showing up in operations

This is not only a coding story. Bain's cybersecurity chapter of the same report notes that AI-powered scanning has pushed vulnerability alerts up by as much as eightfold, and recommends automating simpler fixes "with a human in the loop".

On-call teams know the arithmetic. Google's SRE guidance found that dealing with a single on-call incident, including the postmortem, takes about 6 hours on average, which is why it caps load at 2 incidents per 12-hour shift. Every alert an AI SRE agent investigates still ends with a human deciding what to do. If that human has to repeat the investigation to trust the result, the incident takes just as long as it did before.

What makes a human review fast

The fix is not to remove the human. It is to change what the agent hands back. Bain's respondents rank safety, risk and governance guardrails as the top enabler for scaling agents, at 43%, and 41% expect a risk-tiered model where higher-risk changes go to a human. Good AI agent governance makes review faster, because it tells the reviewer exactly what they are looking at.

Scope before the run

A reviewer moves faster when they already know the boundaries. Microsoft's guidance on governing agents advises teams to grant agents access only to the specific data sources required for their function. When an agent's role, data and tools are fixed in advance, the reviewer does not have to wonder what else it touched.

Evidence with the answer

A finding that arrives with its evidence is a finding you can check. That means the queries run, what they returned and the signals the conclusion rests on. Forrester puts the standard plainly: every autonomous action has to be logged and defensible to an auditor. The same record that satisfies an auditor lets an engineer confirm a root cause without starting over.

Approval sized to the risk

Not every output needs the same scrutiny. Forrester recommends starting agents on bounded tasks behind approval gates and rollback paths, and expanding autonomy as controls prove themselves. An approval request that states the action, how far it reaches and whether it can be undone lets the reviewer decide in one read.

02_after-approval-sized-to-the-risk_risk-ladder.png

Applying it to AI SRE and incident response

Picture a latency alert on a checkout service at 14:25. A bare-answer agent replies "likely a bad deploy". A reviewer then spends half an hour proving it.

An evidence-first agent replies with the deploy event at 14:20, the connection pool metric pinned at its limit since 14:22, the timeout count from the logs and the p95 latency from sampled traces, plus a staged rollback that waits for approval. The reviewer scans four pieces of evidence, checks the scope of the rollback and decides. Same human, same accountability, far less rework.

That is the shift AI incident response needs as agent counts grow: the human stays in the loop, but the loop gets shorter. For a deeper look at splitting investigation from decision, see our post on AI in IT operations, where agents investigate and humans decide.

How Toby AI Agents help

Toby AI Agents are AI teammates that run inside your TrueWatch workspace on your existing telemetry. They are built around the handoff described above.

  • Scope you set. You define each agent's role, data scope and the tools it may use. Roles include SRE, Security, AppDev, FinOps and Custom.
  • Investigation, then findings. Agents investigate, run checks and return findings with proposed fixes. The SRE Agent, for example, reads metrics, logs and traces for its scoped services and delivers a likely cause, the evidence and a rollback or scaling change for approval.
  • Evidence on every run. Each run is recorded with the tools called, what they returned and the evidence behind the answer.
  • Your approval for every change. Anything that changes a system waits for your approval. The request shows the action, its scope and whether it is reversible, with Approve or Decline.
  • Where your team already works. Agents work from dashboards, alerts, DQL queries, Slack or mobile, inherit context such as time range, services and filters, and run on alerts, schedules or questions.

If you want the design thinking behind those controls, read how we designed trust into Toby AI Agents.

03_after-how-toby-ai-agents-help_approval-card.png

FAQ

What are human in the loop AI agents?

They are AI agents that carry out work such as investigation or analysis, then hand decisions back to a person. The human reviews the result and approves or declines any action, especially one that changes a production system.

Why does AI increase review time?

Bain's 2026 survey found developers complete roughly 21% more tasks with AI while review time rises approximately 91%. More output means more to verify, and outputs without supporting evidence force reviewers to repeat the work to trust it.

How does AI agent governance speed up review?

Clear scope, recorded evidence and risk-based approvals tell the reviewer what the agent could access, what it found and what it wants to do. That turns review into a check of evidence rather than a fresh investigation.

Do Toby AI Agents make production changes on their own?

No. Toby AI Agents investigate within the scope you assign and propose fixes. Any change waits for your approval, and each run keeps the tools called, what they returned and the evidence behind the answer.

Delegate the investigation, keep the decision

AI has already made the work faster. The next step is making the review faster, without taking the human out of it. Agents that hand back scoped, evidenced findings and wait for approval keep your engineers in charge while giving them back the hours they spend re-checking. See how Toby AI Agents work.

Get in touch background

Get Started with TrueWatch