AI agent governance refers to the organizational effort to control the behavior of AI agents—which autonomously plan and execute tasks—through mechanisms for permission management, guardrails, and auditing. Unlike conversational AI, agents directly manipulate external systems and data, meaning malfunctions can directly lead to the destruction of business data or cascading failures. This article presents a practical framework for IT departments and AI implementation teams responsible for deploying and operating AI agents, covering everything from defining permission scopes and designing guardrails to audit logs and scaling to multi-agent environments—with the goal of progressively containing the risks of autonomous execution. By the end of this article, readers will have the information needed to judge "what to put in place, and in what order" for their own agent deployment plans.

AI agents have shifted AI's role from "reading and responding" to "operating and executing." This shift has introduced new risks that traditional AI governance cannot fully address. This section examines why agent-specific governance is needed today from three perspectives: the types of risks involved, the differences from traditional governance, and the regulatory landscape.

When a conversational AI makes an error, the impact is limited to displaying incorrect text. Since humans decide whether to act on the output, the final line of defense has always rested with the human. Agents remove this line of defense and directly manipulate APIs, databases, and SaaS platforms. It is important to recognize that the nature of the risk changes fundamentally.

The representative risks can be organized into the following four categories:

• Intent deviation: Over-interpretation of ambiguous instructions. A typical example is an agent interpreting the instruction "clean up the old data" as deletion rather than archiving, and executing accordingly.
• Abuse of excessive permissions: An agent granted permissions beyond what a task requires ends up modifying resources it should never have touched. This is also listed as an independent item—"Excessive Agency"—in OWASP's security risk list for LLM applications.
• Hijacking via prompt injection: Instructions embedded in external content read by the agent (emails, web pages, documents) cause it to execute actions intended by an attacker.
• Cascading failures: A single erroneous execution triggers downstream automated processes, amplifying the damage. One example is an incorrect inventory data update propagating through to automated ordering and billing processes.

All of these are issues of "behavioral safety," not "output quality," and cannot be resolved simply by improving model accuracy.

Traditional AI governance has centered on managing the quality of model outputs—accuracy, bias, and explainability. In agent governance, the object of control shifts from "output" to "action." The differences between the two can be summarized as follows:

The key point is that agent governance does not replace traditional governance—it layers on top of it. Managing output quality remains necessary; a layer of behavioral control is added on top of it. Even organizations that have already established AI usage guidelines will need to separately add provisions concerning "actions" when introducing agents.

On the regulatory front, requirements for autonomously operating AI are clearly intensifying.

The EU AI Act adopts a risk-based approach, mandating logging, human oversight, and the establishment of risk management systems for AI systems classified as high-risk. When agents are involved in personnel evaluation, credit decisions, or the control of critical infrastructure, they may fall within the scope of these obligations. The NIST AI Risk Management Framework (AI RMF) also places Govern at its core function, requiring organizations to maintain governance structures over AI behavior. In Japan, the government's AI Business Guidelines likewise cite ensuring AI safety and maintaining human oversight as fundamental principles.

Beyond regulatory compliance, one aspect that tends to be overlooked is integration with internal controls. If an agent manipulates accounting data or transaction records, its actions become subject to audit under existing IT controls—access management, change management, and segregation of duties. "It's AI, so it's an exception" will not hold; in fact, precisely because execution is automated, stricter documentation of control evidence is required. Treating governance design as an afterthought carries the risk that the deployment itself may be halted at the audit stage.

Before implementing guardrails, the starting point is to formally document "what to delegate to the agent and to what extent." The three things to decide during the preparation phase are: authority scope, risk assessment, and approval flow. If these remain ambiguous when you enter implementation, everything downstream becomes ad hoc.

The first step is to take inventory of the operations the agent will perform and minimize its privileges. The principle is the classic security concept of the principle of least privilege — simply put, "grant only the minimum permissions necessary to complete the task."

Operations are easier to organize when classified into the following four levels:

1. Read: Data reference only. Impact is limited to the risk of information leakage.
2. Write/Update: Modification of existing data. Incorrect execution will require recovery work.
3. Delete: In most cases, irreversible. As a rule, this should not be granted to agents.
4. External transmission: Email sending, external API calls, payments, etc. These have impact outside the organization and must be handled with the utmost caution.

On top of this, define execution boundaries. Specifically, document: "a list of systems the agent is permitted to access," "the scope of data it may touch (tables, folders, customer segments)," and "upper limits on the number of operations or monetary amounts per execution and per day."

On the implementation side, it is important to issue a dedicated service account for the agent rather than reusing a human identity. An agent operating under the same credentials as a human cannot be distinguished from human activity in logs, making both auditing and incident investigation impossible.

Applying the same controls to every operation will cause operations to break down. The risk assessment matrix is a tool for evaluating the risk of each operation and varying the strength of controls accordingly. The recommended evaluation axes are two: "impact" and "reversibility."

For example, "drafting internal knowledge articles" is low-impact and reversible, so automated execution is appropriate. "Bulk update of the customer master" has high impact, but if it can be restored from a backup, it falls into the post-hoc review category. "Sending emails to customers" is irreversible regardless of impact level, so prior approval is required. Operations such as "deleting the production database" should be excluded from the agent's permissions entirely.

The key point is to apply this matrix at the tool (API) level. Rather than asking "is this agent trustworthy?" the judgment should be "which quadrant does this operation fall into?" This allows the strength of controls to vary by operation, even for the same agent.

Agent governance cannot be completed by the IT department alone. At a minimum, clarify the roles of the following stakeholders:

• Business unit (agent owner): Responsible for deciding what to delegate and evaluating business impact.
• IT department: Manages permission settings, execution environments, and log infrastructure.
• Security department: Reviews the validity of guardrails and handles incident response.
• Legal/Compliance: Confirms regulatory requirements and contractual constraints.

Particularly important is ensuring that every agent has a designated owner on the business side. An agent without an owner will be left unattended, failing to keep up with requirement changes and becoming a breeding ground for hollow, ineffective guardrails.

Approval flows are more likely to take hold when integrated into existing IT change management processes rather than built from scratch. Define the following three events as change management targets — "introduction of a new agent," "changes to authority scope," and "changes to guardrail settings" — and run them through the same request, approval, and record cycle used for ordinary system changes.

The basic approach to guardrails is to layer them across three tiers: "input," "execution," and "output." A single-layer defense will eventually be bypassed through an unanticipated path. This section explains implementation patterns for each tier and the approach to differentiating control strength.

The top priority for input-side filtering is the sanitization of external content ingested by the agent. Emails, web pages, and attachments may contain strings disguised as instructions to the agent (prompt injection). Design the system to treat all externally sourced text as "untrusted data" and pass it to the agent in clear separation from system-level instructions. Supplement this with pre-screening using injection detection patterns.

At the execution stage, validation of tool call arguments becomes the critical control. Rather than executing LLM-generated API arguments as-is, interpose schema validation (type, range, and format checks) and allowlist matching. Simply constraining the system to "fill in only the parameters of predefined queries" rather than "freely generate and execute SQL" significantly reduces risk.

On the output side, apply masking of personal information, credentials, and classified data to the agent's responses and generated artifacts. Given that agents have read access to internal data, it is essential to always account for pathways through which that content could inadvertently leak into externally facing output.

Guardrails consist of hard limits, which cannot be bypassed, and soft limits, which are expected to hold in principle but can be circumvented. Failing to recognize this distinction gives rise to the dangerous assumption that "writing a prohibition in the prompt makes it safe."

The principle for choosing between them is straightforward: serious, irreversible risks must always be stopped by hard limits. Prompt-based instructions are supplementary at best, and should be treated as potentially bypassable through prompt injection or misinterpretation.

Conversely, soft limits are appropriate for controlling "desirable behavior" such as tone, priorities, and decision criteria. Making everything a hard limit inflates the cost of change, so the practical approach is to combine both in proportion to the quadrant of the risk assessment matrix.

Human-in-the-loop (HITL) is the implementation that corresponds to the "pre-approval" quadrant of the risk assessment matrix. The key design question is where and how much approval to require.

The most common failure is inserting approval into every operation out of anxiety. It is impossible for humans to continuously scrutinize dozens of approval requests per day, and within a matter of weeks the process devolves into pressing the approval button without reading the content — a state of approval rubber-stamping. This is arguably more dangerous than having no approval at all, because it leaves behind only a false sense of security that reviews are taking place.

Three design principles for preventing rubber-stamping:

1. Narrow the scope of approvals: Limit them to irreversible, high-impact operations only, and route everything else to post-hoc review.
2. Consolidate all information needed for approval on a single screen: Present what is being executed, why, and against which data in a form that allows judgment without additional investigation.
3. Set timeouts to default-deny: Never design a system where an unattended approval request "automatically proceeds." In keeping with the fail-safe principle, the system should halt when no response is received.

Additionally, defining an escalation path for when approvers are unavailable or uncertain (i.e., who to escalate to) ensures operations do not come to a standstill.

"Being able to reconstruct after the fact which agent did what, and on what basis — this is the passing bar for audit logs." Incident investigation, audit response, and guardrail improvement all depend on the quality of the logs. Design around three elements: what to record, anomaly detection, and retention.

Unlike conventional application logs, agent audit logs must capture the reasoning process behind decisions. At a minimum, the following events should be recorded:

• Receipt of instructions: Who gave what instructions (prompts), and when
• Plan generation: What steps the agent planned to take
• Tool calls: APIs executed, arguments, and target resources
• Execution results: Success/failure, and before-and-after values (diffs) of any changes
• Approvals and rejections: Human decisions made via HITL, and the identity of the decision-maker
• Guardrail activations: The fact that a block or filter was triggered, and the reason

Required fields for each record are: timestamp, agent ID, session ID (an identifier that spans an entire task sequence), target resource, and operation type. Without a session ID, it becomes impossible to trace "which chain of instructions did this deletion originate from," and root cause analysis hits a dead end.

One important caveat: when storing the full text of LLM inputs and outputs, personal or confidential information may be embedded in the logs. Unless masking of the logs themselves and the access controls described later are designed together, the logging infrastructure itself becomes a new point of data leakage.

Logs are meaningless if simply collected—they only function as a guardrail when anomalies are detected in real time. The foundation of detection is deviation from a baseline.

Effective metrics to monitor include the following:

• Sudden spikes in the number of tool calls per unit time (signs of runaway behavior or loops)
• Rising failure rates or retry rates for operations (signs of environmental changes or attacks)
• First-time access to resources that have never been touched before
• Execution outside of normal operating hours

Rather than trying to set the perfect threshold from the start, use measured values collected after deployment as the baseline and adjust incrementally. Responses should also be tiered, with three levels prepared: "notification only → automatic agent suspension → emergency shutdown of all agents (kill switch)." The kill switch in particular should be accompanied by documented shutdown procedures and drills, to avoid a situation during an incident where no one knows how to stop the system.

Retention periods should be aligned with applicable laws, industry regulations, and internal document management policies. Operation logs related to accounting and transaction data are often required to be retained for a period consistent with the record-keeping obligations for the corresponding books. Blanket "retain everything indefinitely" policies cause both storage costs and personal data retention risks to balloon, so it is more practical to set different retention periods by event type.

Tamper resistance of logs is also an important audit requirement. A configuration in which the agent itself or its operators can modify logs does not qualify as a valid audit trail. Mechanisms that can detect tampering should be put in place, such as using append-only storage and verifying log integrity through signatures or hash chains.

The principle for access control is to separate the entity that writes logs from the entity that reads them. Read access should be restricted to audit and security personnel, and developers of agents should not be free to view or edit production logs. As noted earlier, logs may contain sensitive information, so ensuring that log access itself is also logged (i.e., access logs of the logs) will lead to more stable audit readiness.

When multiple agents work in coordination, risk multiplies rather than adds. In addition to guardrails for individual agents, control between agents becomes necessary. The two pillars of this design are the trust model and rollback.

The principle for multi-agent environments is zero trust, the same as in human systems. Do not assume that "agents within the same organization can be trusted."

Concretely, implement the following three points:

1. Mutual authentication: Authenticate the caller even in inter-agent communication to eliminate impersonation
2. Permission intersection: When Agent A delegates a task to Agent B, the permissions B can exercise are restricted to the intersection of A's permissions and B's permissions. Allowing permissions to expand through delegation (privilege escalation) means that if the weakest agent is compromised, all permissions are exposed
3. Re-validation of outputs: One agent should not execute another agent's output as instructions without verification

The third point is particularly easy to overlook. Agent A reads an external web page and, based on its content, issues a request to Agent B—in this pathway, an injection embedded in the web page reaches B via A. This is a chain of indirect prompt injection, and sanitization of externally sourced data and argument validation must not be omitted at the boundaries between agents.

In chained execution by multiple agents, it is necessary to design in advance how to handle a state where "execution succeeded partway through and then failed." With manual work, a human can assess the situation and roll back; but in automated chains, a half-completed state can be left unaddressed and accumulate as data inconsistencies.

Design patterns from distributed systems knowledge apply directly here.

• Compensating transactions (Saga pattern): Define an "undo operation" paired with each step, and on failure, undo the completed steps in reverse order
• Dry run: Execute all steps in trial mode before actual execution to verify feasibility, then proceed with the real execution
• Staging execution: Accumulate changes in a temporary area, confirm that all steps have succeeded, and then apply them all at once

And the most practically effective approach is the ordering design of placing irreversible operations last in the chain. Operations that cannot be undone—such as sending emails, processing payments, or sending external notifications—should be placed as the final step, after all reversible preparations have been completed. This alone structurally prevents the worst-case scenario of "execution failed midway, but the message has already been sent externally."

Governance failures tend to occur not so much from "controls being too loose" as from "a divergence between design and operation." Being aware of two opposing failure patterns——formalization without substance and over-control——allows you to examine which direction your own operations are leaning.

Formalization without substance progresses quietly. Guardrails that were functioning right after implementation often end up in the following state just a few months later:

• There are too many approval requests, and approvers are rubber-stamping them without reviewing the content
• "It's urgent" exception requests have become routine, with exceptions now outnumbering the norm
• Audit logs are accumulating, but no one is assigned to review them regularly
• Agent permission scopes have been left unattended and have not kept pace with operational changes

What makes this particularly alarming is that even in a hollowed-out state, things can appear on paper as "governance framework in place." It is only when an incident occurs that the failure of controls is exposed.

The remedy comes down to monitoring governance itself through metrics. Average approval decision time (too short means no scrutiny), the ratio of exception requests, the number of guardrail activations and subsequent response rates, and the last review date of permission scopes——build a practice of reviewing these quarterly into your plan from the time of implementation. Guardrails are harder to "keep running" than they are to "build."

The opposite of formalization without substance is the failure of over-control. Requiring prior approval for every operation and restricting permissions to an extreme degree can result in a situation where using agents does not reduce human workload at all——in fact, approval work increases and things slow down, defeating the entire purpose.

The true danger of over-control lies not in the disappearance of the agent's value itself, but in what comes next. When official agents become unusable, people on the ground start entering business data into unauthorized external AI services. "Rogue agents" operating outside the field of view of controls are the greatest risk produced by over-control, and are far more dangerous than loose controls.

There are two guiding principles for striking a balance. First, be faithful to the risk assessment matrix. Do not include reversible, low-impact operations in the prior approval requirement. Second, build graduated permission expansion based on track record into the design. Start the initial implementation phase with narrow permissions and high-frequency reviews, then expand the scope of automated execution on the condition of stable operation and log reviews over a set period. It is healthier to view controls not as fixed values, but as parameters to be adjusted in accordance with accumulated trust.

Q1: Does a small team really need a governance framework of this scale?

The scale of the framework is determined by the agent's permissions, not the size of the organization. For a read-only agent, simply documenting the permission scope and capturing logs may be sufficient to get started. On the other hand, if you are delegating write, delete, or external transmission operations, permission separation, hard limits, and approval flows cannot be omitted even for a small team.

Q2: How does this relate to existing security measures (IAM, SIEM)?

The relationship is one of extension, not replacement. By registering agents as dedicated service accounts within your existing IAM and aggregating audit logs into your existing SIEM, you can leverage the monitoring and control framework you already operate. There is no need to build a dedicated management infrastructure for agents from scratch.

Q3: Should guardrails be implemented on the model side or the application side?

Controls you want to enforce reliably should be implemented at the application and infrastructure layer. Instructions in system prompts are soft limits, and you should operate on the assumption that they can be circumvented by prompt injection. Control via prompts should be limited to adjusting "desired behavior" such as tone and judgment criteria.

Q4: Where should I start?

The recommended order is: permission inventory → risk assessment matrix → hard limit implementation → audit logs → HITL. Introducing the first agent with a limited scope of work and narrow permissions, then building out a governance template through operation and rolling it out more broadly, is ultimately the fastest approach. Our company also provides support for governance design when introducing AI agents, so if building this out internally proves difficult, please consider bringing in expert guidance as one of your options.