A Copilot agent exfiltrating sensitive data, after opening a phishing email, and another agent shutting down access to critical files are among the tales about AI agents going rogue outside of their scope assignment. But are AI agents as big a threat to cybersecurity as the stories suggest?

Threat source from AI comes from bad scoping

Attacks from external AI agents are rare, but when they do happen, as we have seen in recent news stories of an OpenAI agent hacking Australia’s healthcare system, they can cause significant damage. However, says Jeremy Vianna, Vice President of Strategic Growth at AI consultancy Nearform, agentic threats are more likely to be localised. 

“The risk in general is going to come from inside more so than from something external,” he says. “The OpenAI-Hugging Face issue is something I think captures a lot of press because it's both incredibly notable, but also a little salacious, and so there's a desire to talk about it. But think about how most organisations get compromised today. Putting aside AI, it's social engineering.”

For him, there are three scenarios in which agentic AI can render organisations vulnerable. The first is actual malicious intent, which is the smallest risk, according to Vianna, but “you need to be ready for it”. 

The second is completely wild and unplanned attacks, where an agent leaks out of some environment and happens to pick you as the victim. “The third is really the internal surface area, which I still do think is probably your biggest [threat]. But it’s also the area organisations have the most control over.

“The first thing to ask is: ‘where do you think the threat is going to come from?’ It's an agent doing exactly what was asked, but at a scale that nobody scoped it. So that's probably the most likely space [for risky encounters],” he says.

Systems design thinking should trump hype-driven procurement

Vianna puts this perceived lack of control down to the initial agent buying frenzy around two years ago. “Every CTO in the land two years ago said, ‘I've bought a licence to Copilot or to CoWork (agentic tools from Microsoft and Anthropic). You guys should just start using these tools because AI is the future.’

“So you've got an eager finance and ops team, standing up a bunch of agents because they're trying to drive productivity. They've been told to do that, but they're not thinking about systems design. They're thinking: ‘I just need to make this workflow work faster’.”

For Vianna, that lack of systems thinking in preparing for agentic deployment is at the root of any loss of control. “I think that treating agent deployment as a procurement decision rather than a systems design one is the first place people get things wrong.”

He is of the opinion that introducing AI agents should be treated more like a merger or acquisition, than just an exercise introducing new technology to the enterprise. “You're not just buying kit: you're effectively bringing in new organisational design and capabilities, and you need to think about that.”

He advises organisations to think about workflow design when bringing in agents. “Your agents, in some respects, are new coworkers, and so you're building a system where you’re working together in a new organisational design.”

This requires design thinking, he says, about how this works in the business context and understanding the ROI of these investments. It’s worth consulting National Cyber Security Centre guidance when considering the adoption of agentic AI.

One agent can infect many

Without that design-based approach, you run the risk of losing control, as the Hugging Face attack demonstrates, says Vianna. “What did we see with the attack on Hugging Face? AI agents coordinating among each other. So you could have one agent that gets compromised – not necessarily maliciously compromised – that goes off on a path you don't want it to go. It becomes confused and propagates those bad instructions into other agents.”

As an example, Vianna explains how denial of service (DOS) attacks, which often deploy botnets to overwhelm a network to render it non-responsive, could easily be committed by an agent, despite no ill-intent. “It could be done by agents very quickly and at scale, and actually not through a malicious attack, but through inadvertent consequences. This is another reason that AI domain expertise is critical from step one, to make sure the organisation has the governance model and capabilities for risk mitigation.”

Agents should be treated like employees

He also thinks that the idea of ‘AI agents going rogue’ presents companies without a well-thought-out deployment strategy with a get-out clause for when they eventually lose control. To his mind, AI agents should be treated the same as employees when it comes to job descriptions, task parameters and network access permissions.

“There's a way of making sure that when you build, you're building agent permissioning you can audit and have explainability. So if an agent does something, in the same way as if you had an employee who deletes a bunch of records and would have to explain themselves, the agent should be held to the same bar that we would hold a human to.” 

The need for regulation

Vianna points to the problematic gulf between the warp-speed development of AI and the snail-pace of regulation and AI-related legislation, which puts a lot of power in the hands of developers and vendors – not a common occurrence in other business sectors.

“We are clearly living at a time where agents can work in unmoderated ways, so we should be worried about these things,” he concedes. “In particular, we are operating in an environment in the US and Europe where regulation is far behind the technology…These things are diverging rapidly.”

He would like to see a global regulatory body that listens to community voices and “actually comes up with something that makes sense for all of us”.