Key takeaways
- Safeguards are essential for AI agents.
- Limiting data sources and system access is one important step.
- Make sure you test the agent properly before deploying it.
- Humans need to be involved, despite an agent’s autonomy.
Having done the groundwork to determine the scope, select the model and set the prompts and data sources for the AI agent, it’s important to set safeguards to ensure that the agent does not act in unexpected or unpredictable ways that could potentially impact on company systems and data.
It requires clear processes and the right amount of human involvement to ensure that it can operate safely, says Ciaran Cosgrave, CEO of AI consultancy Nearform.
Prefer to listen?
This audio file was produced by AI and has been adapted from the original article for audio purposes.
“The other ‘silent killer’ is deploying agents with no auditable reasoning; the first untraceable decision, however correct, collapses trust and stalls adoption,” he says.
“In reality, most failures aren’t about model capability; they stem from the organisation not being set up to absorb AI effectively. My advice is always to look at your internal processes before trying to add AI technology as a layer on top.”
Narrow the ‘blast radius’
It’s important to limit the scope of an agent’s operations and access privileges – the ‘blast radius’ of any impact should they go rogue. An agent deployed to summarise and route emails has, for example, no business being afforded database 'write' or ‘delete’ authorisation.
Treat the agent a little like an employee, with clear user profiles and access management. Ensure a very tight scope for the agent.
Ask yourself: does the agent need to access emails or the internet? It may not need access to the same things as your employees.
Sandboxing
When introducing an agent into the workflow, it should be deployed within a secure, often temporary space within the system that houses the agent’s tools.
This means that any hallucination, questionable code generation, rogue logic, or dubious interaction with external data, is contained in an area removed from the host system.
This sandbox should reflect the ‘live’ environment as much as possible.
Nominate your ‘humans-in-the-loop’
For every AI agent deployed, it is imperative that a named human is assigned to take responsibility for:
- monitoring performance;
- the level of access the agent has to the host systems and data;
- the safeguards in place around such access; and
- all actions taken by the agent and the outcomes.
“Organisations need to define exactly what decisions an agent can and can’t make, and critically, who owns those decisions,” explains Cosgrove. “Decision rights are one of the biggest blockers to scaling AI, yet they’re often overlooked,” says Cosgrave.
Once this is dealt with, organisations need:
- human-in-the-loop controls where judgement is required;
- full auditability so decisions can be traced and explained;
- strong data governance; and
- clearly defined failure modes.
Cosgrove adds that there must also be an ability to stop an agent and reverse its actions the moment something looks wrong, baked into the workflow.
Keep backups
If you’ve read some of the horror stories around agents deleting company data or completely wiping inboxes, you’ll know the importance of keeping backups.
Make sure all important files and company data are backed up on disconnected drives that the agent cannot access.
Include a kill switch
As a last resort, it’s a good idea to have a kill switch in place should an AI agent go completely rogue.
This is a control built into the infrastructure of the AI that immediately removes the agent’s access to your systems, such as databases, workflows and APIs.
It sits outside of the AI’s reasoning processes so it cannot be ignored or overridden. There are instructions on how to set up a kill switch online.
Formulate an escalation procedure or policy
If an agent does go rogue and beats any defences you may have in place, think hard about the legal implications for such a breach. For example, does it contravene EU (GDPR and AI Acts) or UK laws? Breach of the former can attract fines of up to €35m, or 7% of global turnover, whichever is the higher figure.
Amsterdam-based non-for-profit Aithos Research Foundation revealed in May 2026 that according to its research “none of the main AI models have acceptable levels of compliance with the EU AI Act and privacy legislation”. So if your organisation is subject to EU law, it’s best to err on the side of caution.
Helpful resources
- National Cyber Security Centre: Think carefully before adopting Agentic AI:
- Careful adoption of agentic AI services (security guidance co-produced by the UK, the US, Canada, Australia and New Zealand):
- The GenAI Divide: state of AI in business 2025 (research produced by MIT’s project NANDA)
- Leading AI Models are consistently breaking the law (Aithos research foundation)
AI agents: initial steps
Before an organisation thinks about using AI agents, due diligence, preparatory work, system piloting and judgement are required. In part 1, we outline four steps to get set up right.
Stay up to date
You can receive regular email updates from ICAEW insights, including weekly or monthly enewsletters. Subscribe to whichever works for you.
Sign up