Current approaches to agentic AI governance seem more focused on trying to apply governance after a system is developed, like a Band-Aid, instead of baking in reasonable governance and controls into the guts of the system. In the same way that security teams refer to “trust assurance” as the measures and frameworks that give them confidence that security controls are predictable, transparent, and working effectively and as intended, these governance-forward principles can and should be applied to reduce AI Agent risks throughout the system lifecycle.
One AI security vendor recently claimed that SOC2 was insufficient for evaluating AI security and controls. Although we might agree with the ultimate conclusion, the vendor made some surprising statements that show an implicit bias for the Band-Aid approach and a poor understanding of how to effectively govern Agentic AI systems.
Here are a few examples of representations that should raise red flags.
Vendor:
“An AI Agent doesn’t have a fixed set of behaviors.”
Correct Approach:
AI agents in most settings should have established, fixed behaviors. Success and failure modes should be explicitly defined and governed. This should be done with static logic (code) and is one of the most important controls in any Agentic AI system.
If success and failure are not well defined and monitored, then arguably the system is constructively allowed to misbehave. And, when it does so, the failure mode will be silent. Trying to detect and stop these system failures after-the-fact with a Band-Aid approach resembles whack-a-mole more than it does governance. Relying on after-the-fact controls is like treating aviation safety as mostly a matter of investigating crashes instead of trying to increase the overall safety and reliability of the system. After-the-fact detection isn’t meaningless, but it is the wrong layer to lean on. In agentic AI, as in aviation, the primary safety work should happen before and during system operation.
Vendor:
“The agent may call APIs that are not part of any predefined workflow.”
Correct Approach:
Workflows should be pre-defined, and AI agents should only call APIs for which they are explicitly scoped in advance and subject to pre-defined static logic/control.
If access to API calls is not limited and controlled, then it becomes easier for those APIs to be accessed either inadvertently or maliciously. Potential exploit chains (which string together multiple agentic components) become deeper and more numerous. To extend the airplane analogy, many commercial airplanes have a control system that sits between pilot action and airplane behavior. Control systems on modern aircraft actively resist or prevent unsafe pilot maneuvers and will limit pitch, bank, and angle of attack to prevent stalls and exceeding safe speeds. These are exactly like the pre-defined limits on API calls: before the system can do the unsafe thing, a control has already prevented that system behavior.
Vendor:
“An AI agent decides at runtime what tools to use.”
Correct Approach:
Although it is true that Agent behavior is dynamic, the vendor leaves open the possibility that natural language (i.e., a call to an AI endpoint) can be allowed to invoke tools. For a system to be reliable, however, tool usage should be controlled with static logic. The reason is because responses from AI endpoints are “probabilistic,” meaning one can’t perfectly control the content of the AI response over time. When tool usage turns on probabilistic responses from AI endpoints then, at some point, they are likely be invoked in ways that make the system unreliable and exploitable. Over time, these small and unexpected responses can chain together and lead to cascading failures. Similar to ungoverned API use, exploit chains become more dangerous and multiply. In other words, when preparing for landing, there should be room for the pilot to make judgment calls based on wind speed, runway length, and physical obstacles, but those judgment calls should only be allowed to flip certain switches in the cockpit.
The Critical Role of Lawyers and Compliance
In the current environment, where Agentic AI risks may not be fully appreciated by existing development and cyber teams, a compliance/procurement/cyber attorney may end up being the first and best line of defense to ensure the organization effectuates real governance and controls over the AI use case.
With respect to every AI system and AI system component, an organization ideally should begin by posing the following questions:
- What am I trusting the system or component to do and why?
- How do I limit trust (and limit risk)?
- What is the system or component allowed to do?
- What is the system or component not allowed to do?
- What is the data the system is acting upon and what are my rights and limitations with respect to that data?
- How does the system enforce the answers to these questions?
- What is the risk of scope creep and how can I monitor the actual use over time?
The goal is to surface those areas where trust or scope of action is unwarranted or unnecessary. The governance process is of course a good deal more involved than this, but this is the right conceptual and practical place to start.