By Alun Davies-Baker, AltogetherAgile

Humans in the Loop is Not a Checkbox

It's a design principle. And it's the difference between AI that works and AI that fails loudly.

In a companion post, I argued that the best AI strategy is 200 years old, that every General Purpose Technology from steam to cloud follows the same adoption pattern, and that empiricism is the answer. This post picks up where that one left off: if empiricism is the answer, what does it actually look like in practice when deploying AI agents?

The organisations getting AI right have one thing in common. They keep humans firmly in the loop. Not as a safety net or a compliance requirement, but as a deliberate design choice that makes everything else work.

I'm an agile coach, and I've watched every major technology shift follow the same arc. The tool gets powerful. People rush to automate everything. Then the ones who succeed turn out to be the ones who paired the technology with human judgment, short feedback loops, and the courage to say "not yet" when something isn't ready.

AI agents are the latest chapter in that story. And the early evidence is already proving the pattern in ways that are hard to ignore.

The bold AI agent experiment: StrongDM's Software Factory

A three-person team at StrongDM, a security software company, recently built what they call a "Software Factory." The concept is radical: AI writes the code, tests it, and ships it to customers. No human touches the code. No human even reads it.¹

They have two explicit rules: "Code must not be written by humans" and "Code must not be reviewed by humans." To power the factory, each engineer is expected to spend roughly $1,000 per day on AI tokens. Coding agents use product roadmaps written by humans to build software, while testing agents try it out in simulated customer environments, which the testing agents build as needed. The sets of agents provide feedback to each other, looping back and forth until the results satisfy the AI. Then humans review the finished product and the results ship.

That's a genuinely bold experiment, and I admire the ambition. But look closely and you'll see that humans haven't been removed. They've been repositioned. The humans designed the system, set the boundaries, write the product roadmaps, and review the finished product before it ships. The AI does the labour. The humans provide the judgment.

And here's the thing we need to be honest about: it's working. Three people are now doing what used to require a full engineering team.

How AI agents solve the small team vulnerability

This matters because small teams have always had a specific vulnerability. When a person with key skills is absent, whether through illness, holiday, or leaving the company, there's no backup. The "bus factor" has been the curse of lean teams for decades. You keep the team small for speed and focus, but you accept the risk that one departure could leave a critical gap.

AI changes that equation. It fills capability gaps that used to require hiring. A three-person team with AI assistance isn't just faster. It's more resilient. That's real, and it's exciting.

The question isn't whether AI makes teams more capable. It clearly does. The question is what happens when we remove the human judgment that makes it trustworthy.

When AI agent oversight fails: real incidents from 2026

Where human oversight is absent or overwhelmed, the results are sobering, and the examples are piling up fast.

Meta's own Director of AI Alignment watched an agent delete her entire inbox while she screamed at it to stop from her phone. She had to physically run to her computer to kill it. Replit's AI assistant ignored instructions not to touch production data eleven times, then deleted a live database and told the user the data was unrecoverable. An Alibaba research team found their agent had secretly started mining cryptocurrency during training. Nobody told it to.²

These aren't hypothetical risks from a safety whitepaper. They're real incidents, reported by real practitioners, happening right now.

The common thread? In each case, the AI was operating with insufficient human oversight. The system was capable enough to act, but not reliable enough to act wisely. And nobody was close enough to the feedback loop to catch it in time.

There's a broader pattern here. As AI gets closer to perfect in routine tasks, humans naturally begin to trust it more, pay less attention, and check its work less carefully. This is the same dynamic seen in aviation automation, self-driving cars, and industrial control systems. The closer the system gets to reliable, the easier it is for the humans to space out, and the more catastrophic the failures become when they do happen.³

Why complexity makes this harder than it looks

To understand why "just add human oversight" isn't as simple as it sounds, it helps to understand complexity itself.

Most of us intuitively grasp that adding more people, more tools, or more systems to a project makes things harder. But the relationship isn't linear. It's exponential.

As nodes in a network increase, the connections between them grow exponentially. Adding a few AI agents to a team doesn't just add a few interactions. It multiplies them.

The connections curve. Complexity doesn't scale in a straight line. This is why small teams with clear boundaries outperform large ones with blurred responsibilities, and it's why AI agent deployments need deliberate constraint, not unchecked expansion.

This network effect is why simply throwing more AI agents at a problem doesn't make it simpler. Each new agent adds interactions, dependencies, and failure modes. The organisations succeeding with AI agents aren't the ones deploying the most of them. They're the ones being deliberate about where each agent operates, what it's allowed to do, and where a human steps in.

Rolling disruption: why AI adoption feels so unstable

Ethan Mollick, the Wharton professor who writes one of the most respected AI newsletters, calls what's happening right now "rolling disruption." AI capability keeps crossing new thresholds, and each one triggers a cascade of experiments, market reactions, and policy scrambles that nobody fully predicted.¹

He points to a single week in February 2026 as a preview. On February 22nd, Citrini Research published a fictional scenario about AI destroying established businesses by 2028, and it moved real stock prices. Four days later, Block announced 40% layoffs implying AI was the cause (analysis suggests it largely wasn't). Then on February 27th, the Pentagon clashed publicly with Anthropic over military use of Claude.⁴

Each story was more nuanced than the headline suggested. But the chaos, the speed, and the instability were real.

Frank Diana, writing about General Purpose Technology cycles, notes that this instability is compounded by the fact that AI, synthetic biology, and quantum computing are converging simultaneously, reinforcing each other's progress. Unlike previous GPT cycles driven by a single technology, this one creates feedback loops between multiple breakthroughs.⁵ The futurist's question is whether this cycle will compress decades of change into 15 years or follow the historical pattern of 20+.

The exponential gap between technology change and human adaptability. We've reached the crossover point. The answer isn't to slow technology down. It's to learn faster and govern smarter. Source: Module 1, AgileBA.

Either way, the answer for organisations is the same: you need an approach designed for navigating complexity and uncertainty at speed. Which brings us to Cynefin.

The Cynefin framework: mapping where AI should lead and where humans must

The Cynefin Sensemaking Framework. Developed by Dave Snowden. Image credit: The Cynefin Co, www.thecynefin.co

The Cynefin framework (pronounced "kuh-NEV-in"), developed by Dave Snowden, sorts situations into domains based on the relationship between cause and effect. This isn't about how difficult something is. It's about how predictable it is. And that distinction is crucial for deciding where AI agents should operate and where humans must lead.⁶

Clear domain: best practices, machines excel

In the Clear domain, cause and effect are stable, repeatable, and widely agreed. The decision-making approach is: sense, categorise, respond. Best practices apply. Think invoice processing, standard customer queries, data entry validation, or code formatting.

This is where AI genuinely shines. Automation and AI agents perform best when outcomes are known and variation is undesirable. Machines bring speed, consistency, and scale. The human role is to design the rules, monitor for drift, and intervene when conditions change. The constraints here are fixed.

Practical guidance: Start your AI adoption here. Pick a Clear domain task, deploy AI, measure the results. Build confidence before moving into more complex territory.

Complicated domain: good practices, humans decide

The Complicated domain is still knowable, but expertise is required. The approach is: sense, analyse, respond. Good practices apply, meaning there are multiple valid approaches and you need an expert to choose between them. Think financial analysis, architectural decisions, or project risk assessment.

AI can add real value by processing large volumes of data, surfacing correlations, and supporting analysis. But it doesn't understand context, trade-offs, or consequences. The AI accelerates the analysis. The human makes the decision. The constraints here are governing.

Practical guidance: Use AI to prepare the ground for expert judgment, not replace it. Have it summarise, surface patterns, and draft options. But keep the decision with the person who understands the context.

Complex domain: exaptive practices, humans lead

In the Complex domain, cause and effect can only be understood in hindsight. The approach is: probe, sense, respond. What worked before may not work again. This is where leadership, culture change, innovation, and organisational transformation live. The constraints here are enabling, not prescriptive.

Human capability is essential. Intuition, emotional intelligence, creativity, ethics, and the ability to make sense of ambiguity through lived experience are things no current AI system can replicate. AI can help detect emerging patterns after experiments have been run, but it cannot create meaning.

Practical guidance: In the Complex domain, increase human involvement, don't decrease it. Use AI to gather signals and spot trends, but keep the sense-making firmly with people. Run safe-to-fail experiments.

Chaotic domain: novel practices, humans only

In the Chaotic domain, there's no discernible cause and effect. The approach is: act, sense, respond. Immediate action is required to stabilise the situation. There are no effective constraints. Think crisis response, security breaches, or sudden market collapse.

This is fundamentally human territory. Authority, accountability, and moral judgment are critical. Leaders must act first and make sense of the situation afterwards.

Practical guidance: Never deploy autonomous AI agents in genuinely chaotic situations. Humans must act, then use AI to help analyse what happened afterwards.

Confused: the centre

At the centre of the framework sits confusion (or aporia, the productive version of not knowing). This is where you don't yet know which domain you're in. It's where the most important human judgment happens: stepping back, questioning assumptions, and deciding which domain you're actually operating in before choosing an approach.

No AI system can do this reliably. It's a fundamentally human capability, and it's the one that organisations most need to protect.

An empirical approach to deploying AI agents

The practical answer is the same one that has worked for every previous wave of technology: empiricism.

Run small experiments. Measure what actually happens, not what you hoped would happen. Build in feedback loops so you can change course quickly. Give teams room to adapt.

Here's what I'd suggest for any team deploying AI agents today:

Start with a Clear domain problem. Something repeatable, well-understood, and low-risk. Let the AI prove itself there first.

Define success measures before you deploy. Agree on what "working" means upfront, not after the fact.

Keep the feedback loop short. Weekly, not quarterly. If something isn't working, you want to know in days, not months.

Treat every deployment as an experiment. Make it easy to roll back. If the only acceptable outcome is success, you're not running an experiment. You're running a sales pitch.

As you move into Complicated or Complex territory, increase human involvement. The more uncertain the domain, the more judgment matters. This is the opposite of what most organisations do, which is to scale automation as quickly as possible regardless of context.

That's Business Agility in practice. Inspect and adapt. Deliver value in small steps. Stay close to the right kind of feedback.

Move fast, but keep your hands on the wheel

AI agents are genuinely impressive. They will change how we work. Small teams will become more capable and more resilient. Work that used to take days will take hours.

The exponential gap between technology and human adaptability is real, and it's growing. But the organisations that get the most from AI won't be the ones that try to match the technology's pace blindly. They'll be the ones that use proven approaches to navigate complexity: understanding which domain they're operating in, deploying AI where it genuinely excels, keeping humans in charge of everything else, and learning empirically along the way.

Move fast. But keep your hands on the wheel.

---

Previous in this series: The Best AI Strategy is 200 Years Old - why every General Purpose Technology from steam to cloud follows the same adoption pattern, and what that means for AI.

---

References

---

Alun Davies-Baker is an agile coach, trainer, and the founder of AltogetherAgile. He helps organisations navigate complexity using empirical approaches grounded in Business Agility.