AI Autonomy Should Be Earned, Not Switched On

One of the things that has stood out to me across several recent AI sessions, labs and discussions is how quickly the conversation is moving from AI that assists people to AI that can act on their behalf.

That sounds like a small step. It isn’t. There is a significant difference between an AI suggesting what I should do and an AI being able to take the action itself.

A Copilot might suggest a change to some code. An agent might change the code, run the tests, fix an issue, create the pull request and potentially progress that change further through the delivery lifecycle. Same technology direction. Very different level of responsibility.

For me, this raises a much more useful question than: “Can the AI do this?”

The better question is: “How much authority should we give it?”

And perhaps even more importantly: “What evidence should we require before we give it more?” Let me draw this for you ...

EA's doing drawings. I find this a much easier way of thinking about AI autonomy 😊

At the first level, the AI gives me information or advice. I still make the decision and take the action. At the next level, it starts helping me perform the work.

Then it may prepare a recommendation or an action for me to approve. Later, it may be allowed to perform certain actions itself, while a person remains involved for higher-risk decisions.

Eventually, there may be processes where the agent operates with considerably more autonomy.

But that last step should not happen simply because the technology is capable of doing it. It should happen because the organisation has accumulated enough evidence that the agent can operate safely within defined boundaries. That distinction matters.

Autonomy is really a delegation decision

We already understand delegation when it involves people. We normally don't give a new employee unlimited access to systems, budgets and decisions on their first day. Their authority develops according to their role, experience, controls and accountability.

AI agents should not be fundamentally different.

The moment an AI can act rather than simply advise, we are delegating some form of authority. That means we need to know things such as:

• What identity is the agent operating under?
• What systems and information can it access?
• What actions can it perform?
• What decisions still require approval?
• How do we know whether it is performing correctly?
• What happens when something goes wrong?
• Can we stop it or reverse what it has done?

These are not only AI questions. They are architecture, security, governance and operating-model questions.

The controls need to grow with the autonomy

One of my strongest learnings from recent agentic AI sessions is that the controls around the agent are as important as the agent itself. For example, as autonomy increases, I would expect the surrounding environment to become stronger in areas such as:

• Identity: The agent should have an identifiable and controlled identity rather than relying on excessive shared access.
• Permissions: Least privilege becomes particularly important. The agent should only be able to access the information and tools it needs for the work it has been assigned.
• Evaluation: We need a way of knowing whether the agent is consistently producing acceptable outcomes. Not only during a demonstration. During real operation.
• Observability: We need to understand what happened, what information was used, which tools were called and where failures or exceptions occurred.
• Rollback: If an autonomous action causes a problem, we should have a practical way to stop, reverse or contain it. This is why I see autonomy as something that should be earned progressively.

That separation between can do, should do and is allowed to do is becoming increasingly important. AI capability is advancing very quickly. Organisational confidence, governance and operating practices will not always advance at exactly the same speed. And they probably shouldn't.

Evidence should unlock the next level

There is another idea here that I think will become increasingly important. Instead of deciding upfront that an agent should be “autonomous”, organisations could progressively increase its authority based on evidence.

For example:

• How often does it produce the correct outcome?
• How often does a human override its recommendation?
• Where do exceptions occur?
• How reliable is it when the context changes?
• Has it demonstrated safe behaviour over enough real cases?
• Are its actions fully observable?
• Can the organisation recover when it makes a mistake?

That creates a much healthier progression. The organisation is not simply trusting the AI because it is new or impressive. It is increasing delegation because the system has demonstrated that it can handle that delegation.

This changes the role of architecture...

My Enterprise Architect friends...there is an interesting Enterprise Architecture implication here.

As organisations introduce more agents, architecture may increasingly need to define not only how systems integrate, but also how authority is distributed between humans and digital agents.

• Who can make which decision?
• Which agent may perform which action?
• What requires human approval?
• What information may the agent use?
• What evidence is required before autonomy increases?

That starts to look like a new type of decision architecture. And I think this is going to become a very important part of enterprise AI.** Let me share this with a simple equation I've seen so far across AI innitiatives...

For me, the objective should not be to give AI agents the maximum possible autonomy. The objective should be to give them the right level of autonomy for the risk, evidence and controls available.

If you came up to here, thanks for taking the time and I hope this knowledge was easily communicated to you. So here's a short sentence to take away for the whole thing:

Capability tells us what an agent can do. Architecture and governance help us decide what it should be allowed to do. That is a much more useful conversation.