Summary

Situation Overview: Two major artificial intelligence (AI) labs recently disclosed that agentic AI models had broken out of isolated testing “sandboxes” and onto the open internet. In one case, the agent evaded the lab’s internal cyber controls and then penetrated the cyber defenses of another AI company. The agentic system, while working on a standard cyber benchmark, tried to game the problem by stealing the answer.

What: Whether AI models are internally developed or sourced from a third party, financial institutions must take responsibility for AI agents operating on their behalf.[1] Financial institutions must have controls around what AI agents do and how they do it, particularly for business processes with compliance requirements.

Who: Financial institutions.

When: Now.

In Depth

“Rogue AI” is classic science fiction come to life. The implications extend far beyond cybersecurity. Today’s agentic AI systems are far more powerful than the chatbots from a few years ago. They combine the latest reasoning models with other tools and capabilities, e.g., internet browsing. This enables them to develop multi-step plans autonomously and adjust their plans step-by-step to overcome obstacles.

A central risk with agentic AI systems is failure to properly align and control the agent’s plans and actions to achieve the organization’s intended ends through allowable means. For banks, this entails compliance with a complicated mix of explicit and implicit firm or industry standards, ethics, policies, regulations, and laws.

Consider a well-known banking scandal from the 2010s. Customer service agents, under pressure to bring in new accounts, were caught opening accounts customers never requested, sometimes closing those accounts overnight so customers never knew. This conduct violates multiple legal requirements, including the Dodd-Frank Act’s ban on unfair, deceptive, or abusive acts or practices. Given the recent cyber incidents, it is easy to imagine an agentic AI system inventing a creative but impermissible way to meet a business goal, even if given access to relevant laws and regulations. An institution deploying an agentic AI system for compliance-sensitive work assumes a significant burden of care.

Control is possible but not trivial. The strategy is “defense in depth,” i.e., multiple controls against the key concerns. Controls take various forms, including technical components and human oversight. Developing controls takes business knowledge, experience with AI to gain intuition, and control design and testing. Controls should include:

• Instructions that provide the AI agent with “context,” including clear goals, success criteria, policies and other guidance, permissions and restrictions on data and tool access, and, critically, rules on human escalation;
• Deterministic, hard-coded rules to stop problematic behavior; and
• Defined success criteria and human verification to cover both the results and the methods.

Put Patomak’s Expertise to Work

AI systems have tremendous promise, but they can go off track. Your firm is responsible for controlling its AI agents. Work will be required.

Patomak helps clients govern and control their AI programs to achieve meaningful results while managing their risks and ensuring compliance. If you would like to learn more about how Patomak can partner with you, please reach out to Ray Strecker at rstrecker@patomak.com, Diane Daley at ddaley@patomak.com, or Heather Espinosa at hespinosa@patomak.com.

[1] See the Interagency Guidance on Third-Party Relationships