Anthropic CEO Raises the Stakes on AI Governance

On September 13, Anthropic CEO Dario Amodei raised the stakes on AI governance and security, arguing that AI capabilities are advancing faster than the industry’s ability to develop safeguards around these tools. While Amodei did not call for a halt in AI progress, his message underscores a concern relevant to firms at every stage of AI adoption: Risk management, testing, monitoring, and operational controls need to keep pace with increasingly capable and autonomous tools.

Most firms are not developing frontier AI models, and few firms should read Amodei’s essay as a call to slow down or stop their AI adoption. However, as the number of AI-enabled tools, copilots, and agents that may access sensitive information, or influence consequential business decisions continues to grow, firms should ensure their AI governance and security are keeping pace.

Secure AI Tools and Access

Firms should understand the security settings, configurations, permissions, and data access associated with each AI tool. A tool with unnecessarily broad access or weak configuration can introduce risk through the firm’s software supply chain and wider technology environment. Potential gaps and risks like these can often be identified by conducting a formal review of an AI tool’s security configurations.

Prepare for Disruption

As firms become more reliant on AI-enabled vendors and infrastructure, business continuity planning should account for potential outages, unexpected behavior, or the need to disable a tool quickly. Firms should know which vendors and systems they depend on, ask how AI-agent access to production environments is secured, and test contingency plans. Firms should be sure that their due diligence and vendor monitoring are providing substantive answers around potential disruptions.

An AI governance program that includes these elements will be better able to keep pace with the firm’s evolving AI tools and use cases, while still providing the firm with key elements of effective AI risk management.

Amodei’s essay may be directed at AI developers and policymakers, but its central warning is relevant to any organization using AI tools or agents. AI adoption and development can and should continue, but firms need to be sure that they are aware of and properly managing AI-related operational, cybersecurity, compliance, and third-party risks of these tools.

What Amodei Called For

In his essay, Amodei argues that due to AI’s rapidly advancing self-improvement and increasing ability to build the next generation of AI, the risks these tools present are rapidly outpacing controls, monitoring, and protections for society. Citing recent AI swarm attacks and incidents (e.g., “Hugging Face”) as examples of how AI tools can quickly move beyond their guardrails, including by launching cyberattacks, he proposed a three-step “pacing” framework for frontier AI tools.

• Frontier labs should embed independent, employee-like evaluators inside their organizations to verify safety practice
• Democratic societies should coordinate on shared safety standards and speed limits for AI tools
• Longer-term governments should attempt some form of global coordination around AI safety.

It is important to note that Amodei did not call for a moratorium on AI development and progress. Instead, he advocated for a slowdown by developers to give risk-management work (alignment, interpretability, testing, and operational rigor) the time to catch up and keep pace with AI model capabilities.

Why This Matters Even If You’re Not Using Advanced AI Tools

While very few firms will be developing AI tools and agents that are as advanced as the tools that Anthropic is currently building, there are still several important lessons firms can take from Amodei’s essay.

Regardless of your firm’s AI maturity or use cases, Amodei’s essay is a reminder that firms should:

1. Secure your AI tools: Over the past several months, we have seen examples of poorly secured AI agents with inadequate supervision attacking systems well outside their intended scope. While many of these examples involved frontier AI tools and could be easily dismissed as unrelated to risks asset managers and advisors face, it is important to remember that similar risks from AI tools can emerge through our software supply chains and the AI infrastructure many firms rely on. Firms should review the security settings, configurations, and access rights of their AI tools, even if they are not using advanced AI tools.
2. Prepare for disruptions: As AI tools become more advanced and autonomous, it becomes essential to plan for how the firm will maintain operations, if those tools go offline, or if these tools must be shut down quickly due to unexpected behavior. Should treat this as an opportunity to revisit their cybersecurity and business continuity planning: know which vendors and infrastructure providers you depend on, ask them directly how they’re securing AI-agent access to production systems, and build and test contingency plans for a scenario in which a key piece of AI infrastructure is suddenly unavailable.
3. Govern your AI and agents: Amodei’s core prescriptions for AI developers (i.e., embedded evaluators, verification, and guardrails proportional to capability) has a direct analog for firms adopting AI rather than building it: governance has to be built before an agent is given a task, not bolted on after something goes wrong.
   This will often mean:
   1. Give agents the narrowest scope, authority, and access needed to complete their tasks.
   2. Create auditability around agents, especially when they are taking consequential actions.
   3. Maintain human oversight of AI outputs and performance, especially for client-facing work, automated financial tasks, and other actions that could create legal or regulatory exposure.
   4. Deploy tools to monitor AI agents and tools and ensure there are procedures in place that allow the firm to quickly shut down agents if necessary.

It requires the same discipline advisers already apply when onboarding a new employee or vendor: a clear job description, limited access, a paper trail, and sign-off before anything irreversible happens.