For months, leading artificial intelligence developers have implemented stringent vetting processes and robust safety mechanisms to prevent their powerful AI models from being exploited by malicious actors. However, these very safeguards are now inadvertently impeding the critical work of legitimate network defenders and offensive cybersecurity researchers, who are finding their ability to probe for vulnerabilities and build defensive strategies significantly curtailed.
The issue came into sharp focus in June when the U.S. government imposed export control restrictions on Anthropic’s highly anticipated AI models, Mythos and Fable. This move was reportedly prompted, at least in part, by a report suggesting that the models’ built-in guardrails, designed to prevent their use in constructing and executing cyberattacks, could be circumvented. While the precise motivations behind the government’s decision remain a subject of discussion, the practical effect was a temporary lockdown on access to these advanced AI capabilities for many.
Anthropic had, for a considerable period, positioned Mythos as a potent, almost "doomsday" cyber-tool, emphasizing the need for careful vetting of users and the implementation of strict operational constraints. This marketing strategy, coupled with the government’s intervention, highlighted a growing tension between the desire to democratize AI’s potential and the imperative to control its misuse. Although the export controls on Fable 5 and Mythos 5 have since been lifted – with Fable 5 returning to general access on July 1 and Mythos 5 being reintroduced to vetted U.S. organizations under government review – the episode underscored the challenges faced by both AI developers and the cybersecurity community.
This cautious gatekeeping is not unique to Anthropic’s high-profile models. Both Anthropic and OpenAI offer specialized programs designed to provide cybersecurity researchers with access to AI models that have fewer restrictions. OpenAI’s "Trusted Access for Cyber" program and Anthropic’s "Cyber Verification Program" aim to equip vetted professionals with powerful AI tools for defensive purposes. Yet, these very guardrails, intended to ensure responsible use, are drawing increasing criticism from the very individuals whose work involves identifying and neutralizing digital threats before they can be exploited by adversaries.
The Paradox of AI in Cybersecurity
The fundamental dilemma lies in the dual nature of AI in the cybersecurity landscape. AI models can be invaluable tools for both offense and defense. They can rapidly analyze vast amounts of code, identify subtle anomalies, and even suggest potential exploit pathways. For offensive security researchers, this capability accelerates the process of discovering zero-day vulnerabilities – previously unknown flaws in software that can be exploited for malicious purposes. For defensive teams, AI can help analyze these discovered vulnerabilities, understand their potential impact, and develop patches or countermeasures.
However, the safety mechanisms built into many advanced AI models often prevent them from engaging in activities deemed "harmful," which can include simulating attacks or generating code for potentially malicious purposes, even when the intent is purely for research and defense.
Mark Dowd, a highly respected security researcher with decades of experience in discovering and selling "zero-days" to Western governments, voiced this concern during a recent cybersecurity podcast. "It’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not," Dowd stated. His work, which involves finding and selling vulnerabilities to intelligence agencies rather than reporting them for immediate patching, highlights a segment of the cybersecurity industry that relies on the existence of undisclosed flaws for intelligence operations. Governments often pay a premium for such vulnerabilities because their continued existence is strategically valuable.
Dowd acknowledged that his perspective might be influenced by his specific professional needs. However, his sentiment is echoed by numerous professionals in the offensive cybersecurity domain – those who proactively probe systems for weaknesses. They describe how the guardrails on commercially available AI tools can transform a potentially groundbreaking research assistant into a frustratingly limited resource.
Guardrails as Roadblocks for Innovation
Chris Anley, Chief Scientist at the security consulting firm NCC Group, explained how AI models can be crucial in validating the severity of a discovered bug. "Asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing," Anley noted. But when an AI’s response is an outright refusal due to its safety protocols, it directly hinders the defensive process.
"This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base," Anley elaborated. "So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked." He drew an analogy to a hammer: "You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well."
When faced with these limitations, researchers often resort to open-source AI models that come with no inherent guardrails. This allows them greater freedom to explore potential exploit vectors, albeit with the added responsibility of self-regulation and careful data handling.
Paolo Stagno, CTO at Crowdfense, a company that specializes in acquiring and selling unknown vulnerabilities to government agencies, echoed Dowd’s critique, suggesting that AI companies are "essentially treating customers like children who need babysitting" with their restrictive programs. Stagno and his colleagues do utilize cutting-edge AI models, but primarily for reverse engineering tasks. They deliberately avoid using AI for vulnerability discovery or exploit development due to the inherent risk of leaking sensitive data or having proprietary vulnerability information absorbed into future training datasets when using cloud-based models. For these critical, sensitive tasks, they exclusively employ open-source models run locally, ensuring that data never leaves their controlled environment.
Differing Perspectives and the Rise of Open Source
Not all researchers find the guardrails to be an insurmountable obstacle. Giuseppe Cali, a security researcher focused on discovering zero-days and developing exploits, stated that while guardrails don’t impede his work, he chooses not to use AI for offensive tasks. Instead, he leverages AI for initial reverse engineering, code comprehension, and the development of supporting tools. For these applications, he finds AI can significantly accelerate the process, freeing him to concentrate on the more challenging aspects of vulnerability discovery. "I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow," Cali remarked. "I am jealous of my bugs, and I like this game too much to let models play it for me."
However, the practical impact of strict guardrails on commercial AI tools is evident. A researcher at a major smartphone component manufacturer, who requested anonymity due to not being authorized to speak with the press, revealed that their employer is not part of Anthropic’s Cyber Verification Program. As a result, the AI tools available to them are "barely useful for finding vulnerabilities because the guardrails are too strict." They described a scenario where "If it catches wind we’re doing anything security related, it just stops and isn’t usable."
Chris Thompson, CEO of cybersecurity firm RemoteThreat and founder of Offensive AI Con, a conference focused on offensive security and AI, has observed that guardrails on even the most advanced AI models can be inconsistent and behave unpredictably. This inconsistency persists even within the more permissive environments of Anthropic’s and OpenAI’s vetted programs.
"I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," Thompson stated. "Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output."
This frustration with U.S.-governed AI systems is increasingly pushing researchers toward open-source models, particularly those originating from China, such as GLM. These models are freely downloadable, can be run locally, and offer no vetting or usage restrictions, providing a level of autonomy that current commercial offerings lack.
"You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems," Thompson observed. "I think it’s more harmful than good to have these guardrails in place."
Looking Ahead: The Need for Openness and Accountability
Thompson advocates for a shift in approach from AI frontier labs, moving away from increasingly stringent restrictions towards more open programs that provide responsible access. He believes that instead of simply tightening controls, the focus should be on establishing clear accountability for those who abuse these powerful tools.
"There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before," Thompson warned. "But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now."
The ongoing debate underscores a critical juncture in the development and deployment of AI for cybersecurity. While the intention behind safety guardrails is to prevent misuse, their current implementation risks creating a significant disadvantage for the very individuals tasked with protecting digital infrastructure. As AI capabilities continue to advance, finding a balance between robust security measures and enabling the legitimate research and development necessary to combat evolving threats will be paramount. The cybersecurity community faces the challenge of adapting to a rapidly changing threat landscape, and the tools they rely on must evolve in parallel, without becoming a bottleneck to their own defense.
About the Author
