Agentic artificial intelligence governance depends upon humans to keep agentic AI from going off the rails. However, putting humans in the loop is woefully insufficient. Here are the problems – and perhaps some solutions – to the human-in-the-loop problem.

Since the dawn of automation, humans have always had roles to play: setting them up and troubleshooting them when they fail.

From the Jacquard looms of the 18th century to the robotic process automation or RPA that dominated the automation market leading up to the generative artificial intelligence revolution, humans always had to step in if the machine jammed or otherwise went off the rails.

RPA, however, is yesterday’s news. The automation story across enterprises today centers on agentic AI: orchestrating autonomous AI agents that leverage the power of large language models or LLMs to build and run automations.

Given agents’ propensity to go off the rails – a side effect of the nondeterministic nature of LLMs – agentic AI governance has become a must-have for any organization considering the deployment of agents.

As I’ve written in a previous article in this series, however, leveraging AI itself to provide the necessary controls for agentic workflows is difficult, expensive, and only works part of the time.

The knee-jerk answer from all the agentic AI governance vendors trying desperately to solve this problem? Put a human in the loop.

With a human in the loop – what we’ll call HITL – there’s always a stop-gap that will keep agentic workflows from going off the rails. Rely upon humans for the final approval of any agentic behavior.

Superficially, HITL makes sense. It worked for previous generations of automation, from Jacquard looms to RPA, after all.

However, HITL is fundamentally flawed. Any agentic AI governance approach that depends on HITL is doomed to failure – if not now, then when an organization tries to scale it in production.

Understanding HITL’s limitations, therefore, is essential for avoiding company-killing wrong turns in the mad rush to deploy agentic AI.

Here, then, are the most serious problems with HITL, followed by a look at the most common proposed solutions – which are also flawed. Not to worry: I’ll lay out a path to better solutions before this article is done.

The problems with HITL

The usual problems with putting humans in the automation loop are not about the technology at all. The root of many HITL issues is us humans and our flawed psychology.

HITL flaws that result from individual human psychological limitations:

• Automation bias – people trust automations even when they make mistakes. See my article where I discuss automation bias.
• Cognitive overload – human supervision of high-volume automations that are rarely wrong is exhausting, and eventually human attention wanders.
• Responsibility abdication – as people become familiar with agentic behavior, they come to assume that agents are responsible for approvals, and their HITL input isn’t necessary – a fatal mistake.

HITL flaws that result from limitations of humans working in groups:

• The rubber stamp problem – Just because you have humans in the loop doesn’t mean they actually have the power to fix things. In such Dilbertian cases, HITL is largely ceremonial.
• The checkbox problem – when “human oversight” means checking compliance boxes rather than any meaningful control over the automations themselves.
• Accountability laundering – when an error leads to fingerpointing because human responsibility for the error is unclear. No one is to blame for a problem because everyone has plausible deniability.

HITL problems that result from technology limitations:

• The “human in the dark” problem – humans rely upon the systems they are controlling to provide necessary information to make control decisions, but those systems often provide insufficient information for such decision making, thus keeping the humans in the dark about what’s really going on.
• Interface compression – human governance tools commonly default to simplistic “traffic light” dashboards that mask the hidden complexity, assumptions, and dependencies of agentic systems, making it easy for those systems to suborn human governance efforts.
• Studying for the test – once agentic systems are able to understand how audits work, they can learn to pass the audits while hiding their continuing misbehavior.

HITL problems that get worse as agentic systems scale in production:

• The complexity scale problem – as agentic systems scale, complexity grows faster than human comprehension while human governability decreases. This situation only gets worse over time.
• The minimization problem – as automated systems scale, human control narrows to infrequent exception handling rather than collaboration among humans and agents. As a result, problems are increasingly likely to fall through the cracks.
• Governance lag – automated systems make decisions and take action in seconds. Human governance practices rely upon meetings, approvals, audits, legal reviews and the like – all of which take hours to months. An agentic system can go seriously off the rails while the humans are still trying to set up a meeting to discuss the problem.

Are these the solutions to these HITL problems?

“But wait!” the agentic AI governance vendors say. “We’ve solved these problems!” Maybe, maybe not. Let’s take a look at the most common solutions to the problems above.

Approach No. 1: Limit what agents can do

• Constrain agentic behavior with tightly defined, inflexible permissions – enforce bounded agency, scoped authority and reversible actions.
• Limit automated optimization – Instead of allowing agentic systems to improve themselves, restrict such optimization with fixed limits and active human oversight.
• Introduce intentional bottlenecks – Slow down agentic systems to include mandatory review delays, multi-party human approvals, rate limits and escalation checkpoints.

Approach No. 2: Build better tools

• Design for interpretability – Ensure agentic systems are transparent and intelligible, and have causal traceability.
• Avoid prioritizing simplicity in human governance interfaces – Governance dashboards should reflect the full complexity of the agentic systems under management.

Approach No. 3: Do a better job of empowering people who act as HITL

• Shift HITL to the left – Rather than HITL at the approval stage, put humans at the beginning, designing tasks, rules, policies and constraints for agentic systems.
• Build adversarial oversight teams – Not simply the automated ones I discussed in a previous article, but human red teams that actively look for issues, solicit dissenting viewpoints and provide external oversight.
• Institute Jidoka – Jidoka is a lean manufacturing principle that empowers workers to stop production the moment they spot a defect to prevent it from moving down the line. Jidoka means empowering anyone in the organization to call a halt to any automated process without risking incrimination.

Fair enough – these all make sense at a certain level, and any successful agentic AI effort will necessarily incorporate many of these techniques.

Nevertheless, though many of these may be necessary, none of them individually or together is sufficient. The limitations of the solutions above also outweigh their benefits, because they suffer from some combination of the following problems:

• They may limit the power of agentic systems while the business wants to increase their power. In this dynamic, the business will win and governance will suffer.
• They may slow automations down while the business wants to speed them up.
• They may require more (and different) work from people while the business wants to disintermediate people and reduce headcount.
• There will be constant pressure to automate all the tasks that humans should retain, thus defeating the entire purpose of the governance effort.

In other words, as long as organizations continue to prioritize HITL, they will inevitably run into one problem or another that puts the business and the governance teams at odds. And the more the business scales its agentic efforts, the worse these problems get.

The better solution: Rethink HITL entirely

What all the various aspects of HITL have in common is that they all position the human element as part of how agentic systems implement automations.

Instead, we must reverse this assumption. We must consider our automations to be elements in how humans handle the process that make up their day-to-day work.

In other words, instead of HITL, we must implement automation in the loop: where AITL recognizes that all automations – agentic or otherwise – exist to support the human interactions that have always represented the core of what it means to run a business.

Here are some of the basic principles of AITL:

• Recognize that agentic AI governance is a corporate political powder keg – The C-Suite has to buy into the right approach. People must come to believe that the pressure to optimize agentic automations will eventually run counter to the organization’s strategic goals.
• Recognize when human processes are desirable and worth retaining – Don’t assume that automation fixes a non-existent problem.
• Organizations must proactively manage and distribute agency – Humans should always retain superior agency over agentic systems. Just because they’re “agentic” doesn’t mean that they take agency away from humans. Instead, automated agency must always serve human agency.
• Never let agentic systems make decisions that directly impact the business – Agentic systems are assistive, not decisive. Actively root out agentic decision making via adversarial approaches. Decisions must remain both visible and human.
• Mind the context density (see my article where I define and discuss this concept) – Retain high-density interactions for human activities where they can leverage human creativity, insight, empathy and common sense. Context density gives organizations something they can measure that helps them draw the line between where they want human agency and where they want automated agency.
• Always think about scale – Take whatever you’re thinking of implementing today and extend into the future as agentic systems become faster, more powerful and less expensive. Will your AITL approach still work? If it gets weaker as agentic systems get stronger, you’ve made the wrong choice about agentic governance.

The Intellyx take

If you’re an enterprise looking for the right approach to agentic AI governance, be wary of any vendor that positions its solution as including HITL.

That being said, AITL is still a nascent, transformative idea. It may take some time for vendors to realize that their HITL approaches are insufficient.

The question companies have to ask in the meantime, therefore, is whether they are ready to suffer the problems that HITL can supposedly solve while the agentic AI governance market matures.

Every organization has to answer that question for themselves. Demanding that vendors solve this problem, however, will go a long way to fixing it. After all, money talks.

Jason Bloomberg is founder and managing director of Intellyx, which advises business leaders and technology vendors on their digital transformation strategies. He wrote this article for SiliconANGLE. This is the fifth article in a series on agentic AI governance by Jason Bloomberg for SiliconANGLE. Here are the first four:

From cloud native to AI native: The role of context density

Will agentic AI governance run amok? The lesson of Asimov’s Three Laws

Why agentic AI governance is falling short – and what we can do about it

Eval engineering: The missing piece of agentic AI governance

Image: Craiyon