AgentOps is a framework of "organizational design + operational infrastructure" for keeping AI agents running stably.

While DevOps addressed "operations for safely releasing code" and MLOps addressed "mechanisms for stably operating models," AgentOps deals with "the organization and mechanisms for continuously operating multiple autonomously executing agents across three axes: quality, cost, and compliance." It may appear to be a straightforward extension of LLMOps, which emerged with the advent of LLMs, but the moment the premise that agents "autonomously call tools" is added, the target of operational design shifts from inference pipelines to "business processes themselves."

This guide is intended for DX / IT department heads at mid-sized enterprises, and organizes the components of AgentOps, its integration with existing operations, and implementation steps. By the end, readers should be in a position to draft a responsibility matrix defining "who monitors what, and against which SLIs/SLOs," and to determine priorities for moving pilot-stage agents into production deployment.

AgentOps refers to the concept and infrastructure for managing multiple AI agents as "formally operated workloads within an organization." Rather than a mere tool adoption, it centers on organizational preparation that combines the design of ownership, observability metrics, and human intervention.

This section organizes the differences from existing Ops disciplines (DevOps / MLOps / LLMOps) and explains why agent operations introduce new challenges. Clarifying "what needs to be added to existing operations" is the starting point for building an implementation plan.

The following table compares the areas of concern across traditional Ops disciplines and AgentOps.

What makes AgentOps distinct is the added premise that LLMs autonomously call tools. Up through LLMOps, the primary concerns were "the quality and speed of text generation," but agents write to internal databases, send payments via external APIs, and delete files. Because actions with side effects occur on a routine basis, post-deployment operations cannot be reduced to simple latency monitoring.

Three triggers for the transition are commonly observed: (1) the point at which agents begin issuing actions with side effects to external APIs, (2) the point at which multiple agents are configured to access the same data concurrently, and (3) the point at which "data source-derived prompts" beyond user input enter the system. Once all three are present, operations built as an extension of LLMOps begin to break down.

The nature of agents "making autonomous decisions" introduces three new operational challenges.

• Normalization of non-determinism: Even with identical inputs, the order and selection of tool calls can vary, reducing reproducibility. Bug reproduction becomes difficult, and isolating root causes requires an approach of "re-executing N times with the same input."
• Unpredictable cost variability: Token consumption per task fluctuates significantly depending on "how many steps of reasoning are performed." Since costs for the same feature can vary several-fold depending on user input, budget management based solely on averages is prone to failure.
• Ambiguity of accountability boundaries: When a judgment made by an agent turns out to be incorrect, if it has not been defined who bears responsibility—the providing team, the using department, or the administrative division—response efforts stall. Incident reporting workflows can end up requiring discussion from the very question of "which team files the ticket."

These issues cannot be resolved by individual tools alone; they must be absorbed through organizational rules and role definitions. This is why AgentOps encompasses "organizational design." When any of these three challenges surfaces in the field, establishing escalation paths and owner definitions before selecting tools will, in practice, yield faster results.

As long as a single agent is running for a single purpose, operations can remain lightweight—but the moment the number of agents grows, operational burden expands exponentially. The field observation that "pilots succeeded but scaling hit a wall" is driving growing interest in AgentOps.

This section breaks down what "hitting a wall at scale" actually entails, and explains why traditional operational design cannot absorb it.

The "think → tool call → result" loop that was traceable with a single agent gives rise to multi-agent-specific operational problems when multiple agents collaborate:

• Which agent called which tool, and in what order
• When a failure occurs midway, from which point it is safe to re-execute
• Conflicts arising from concurrent execution (e.g., two agents updating the same record)
• Whether context is lost during handoffs between agents

The fact that frameworks like Mastra and LangGraph are moving toward "describing agent state as a workflow" is itself a response to reclaiming traceability. To prevent a situation where "no one knows what happened," it is necessary to record all inputs, outputs, and reasoning behind each agent's decisions in structured logs, and to maintain a state where workflows can be replayed as a unit.

The MCP-based autonomous execution covered in the Lao Language AI Agent Implementation Guide faces the same operational problems at the production stage. Even if implementation patterns are in place, falling short on operational infrastructure leads to a situation where the system "runs but cannot be stopped."

Here is an overview of the three capabilities that become essential in agent operations.

These three can be implemented independently, but designing them to reference one another reduces operational overhead. Examples of such integration include: using the audit log to determine whether HITL is required, enforcing HITL when costs exceed a threshold, and merging HITL approval logs into the audit log.

Cost management in particular is an area where the small numbers observed during the Pilot phase can balloon significantly upon production rollout. The causes generally fall into three categories: (a) the number of users increases, (b) the number of inference steps per task increases, and (c) prompts grow large due to long-context RAG. It is necessary to define cost measurement units and alert thresholds before introducing any tools. Agreeing in advance on the operational policy—whether to automatically stop the agent or switch to HITL approval when a threshold is exceeded—enables faster decision-making in the field.

AgentOps consists of five elements: (1) agent registry, (2) observability (SLI/SLO and cost), (3) HITL and escalation, (4) evaluation loop, and (5) governance policy. This section takes a deeper look at the first three, which are of particular importance.

The evaluation loop refers to "a mechanism for continuously re-measuring agent quality," while the governance policy refers to rules governing "which agents are permitted to handle which data." It is more efficient to tackle these after the first three elements are in place; attempting to address all five simultaneously from the start risks ending the pilot without a solid foundation.

Once the number of agents exceeds around five, it becomes unclear "where each agent is and who is responsible for it." A registry should hold the following minimum information:

• Agent ID and purpose (customer support / accounting / sales assistance, etc.)
• Owner (business unit owner) and technical contact (developer)
• List of MCP / tools used
• Accessible data categories and PII classification
• Version and change history
• Planned deprecation date (if applicable)

An internal wiki or Notion is fine, but if the operational rule of "always update this whenever a change is made" is not followed, situations arise during audits where agents that no longer exist are still listed. Since maintaining a registry in a correct state carries non-trivial operational overhead, building in a mechanism to automatically push updates from the deployment pipeline (e.g., reflected via PR when updated in CI) from the outset helps prevent the registry from becoming a formality.

SLIs (Service Level Indicators) for AgentOps require a different perspective than latency and error rates for web services. Here are four metrics to monitor at a minimum:

• Task success rate: The proportion of tasks completed without HITL intervention
• Average number of tool calls: How many times tools are invoked per task (an increase is a signal of reasoning going astray)
• Token cost / task: Total of input + output + cache consumption
• Escalation rate: The proportion of cases escalated to HITL

SLOs (target values) should not be set too strictly from the start; measure actual performance during the Pilot period before defining them. For example, committing to "a task success rate of 90% or higher" from the outset tends to push the operations team toward dressing up results, obscuring the improvements that actually need to be made. If the success rate during the Pilot is around 70%, starting the SLO at 75% and raising it incrementally allows the organization to run a more realistic improvement cycle.

An anti-pattern to avoid is designing a system where only the error rate is used as an SLI. Agents can fail in ways that error rates do not capture—such as "consuming cost through wasteful reasoning loops without actually failing" or "operating correctly but producing results that differ from user expectations." Combining task success rate with average number of tool calls enables earlier detection of signs of quality degradation.

Designing HITL is the work of finding a middle ground between "involving a human in every request" and "leaving everything to the agent." Three categories of patterns are used selectively.

• Always-required HITL: Money transfers, contracts, legal judgments, external announcements (domains where the impact of a misjudgment is irreversible)
• Threshold-based HITL: Confidence score < N, amount > ¥N, number of files deleted > N
• Sampling HITL: Auto-approve while randomly routing a fixed percentage to human review (ongoing quality monitoring)

The practical approach is to combine all three within the same agent. For example, an "expense reimbursement agent" might be designed so that transactions over ¥100,000 always require HITL, those under ¥10,000 are automated, and 5% of everything in between is randomly routed to human review.

For escalation targets, it is useful to maintain three channels: "a dedicated reviewer team," "business unit owners," and "a security team," enabling routing to the appropriate person by domain. Defining response time SLAs for reviewers also prevents HITL wait times from becoming a bottleneck for the overall workflow, and makes it easier to design timeout logic on the agent side.

The most common misconception is treating AgentOps as something that can be implemented simply by purchasing a dedicated tool. In reality, the majority of the work involves establishing organizational roles and decision-making structures, and tooling is just one element that supports them.

Here we address two particularly common misconceptions and explain concretely why each one stalls organizational progress.

With the rapid proliferation of AgentOps SaaS products, the perception that "adding an observability tool means AgentOps is done" has become widespread. Observability tools are certainly useful, but on their own they cannot resolve the parts that require human judgment, such as:

• Which SLIs to classify as "unacceptable"
• Who handles cases escalated to HITL
• Who has authority to raise the budget cap when a cost-overrun alert fires
• Who has the authority to shut down an agent when an incident occurs

A dashboard full of red alerts with no one acting on them is the classic failure mode of leading with observability tooling. The priority, in parallel with tool adoption, is to establish escalation paths and a decision-authority matrix. At a minimum, summarizing "who looks at what, at which threshold, and contacts whom" on a single page and reaching agreement across relevant departments before ordering the observability tool is, in the end, the fastest route.

Announcements such as "we have deployed 50 AI agents" appear both internally and externally, but headcount is a poor proxy metric for value. Ten agents may be sufficient in some cases, and it is not uncommon for only three to actually be used. It is also an observed phenomenon that fewer than ten of the fifty deployed agents are active in a given month.

Effective proxy metrics for value include:

• Measured reduction in working hours (by department)
• Trends in HITL volume (a decrease indicates growing proficiency; an increase indicates new use cases being explored)
• User satisfaction (NPS or retention rate)
• ROI (hours saved × labor cost − operating cost)

Agent count is meaningful for budget allocation and headcount planning, but using it directly as a KPI creates the trap of making "increasing the number" an end in itself. It is worth distinguishing how the metric is used in internal proposals. For executive reporting, centering the narrative on "reduction in working hours" and treating agent count as a supplementary reference figure within the breakdown makes it easier to avoid conflating means with ends.

The practical approach to adopting AgentOps is to assign one person from the existing SRE/DevOps team and gradually carve out dedicated roles from there. The approach of standing up a dedicated new team typically fails due to challenges in talent acquisition and friction with existing teams.

Here, assuming a mid-sized company (an IT department of 5–30 people), we organize a realistic adoption path from two perspectives.

Existing SRE / DevOps teams already have observability infrastructure, incident response processes, and on-call rotations in place. Extending these to AI agents is the fastest path forward. Specifically:

1. Assign one person within the SRE / DevOps team as the "AI workload owner" and formally document their scope of responsibility
2. Ingest MCP / agent logs into your existing SIEM / observability tools (JSON Lines format is easiest to work with)
3. Add a dedicated "AI anomalies" chapter to your existing incident response runbooks, explicitly covering shutdown, restart, and rollback procedures
4. Include AI agents in your existing on-call rotation (though for the first few months, only page the AI owner, then expand to other members once sufficient knowledge has accumulated)
5. Share AI workload utilization rates and incident counts at an all-hands SRE meeting every six months

The key point is not to create a dedicated AI department. Following the same philosophy as internal AI assistant adoption, the goal is to expand scope while maintaining operational continuity. Forming a dedicated team is worth considering only after the organization has grown and AI workloads exceed 30% of total SRE capacity — anything earlier is difficult to justify from a human resource perspective.

For mid-sized companies, the required roles can be narrowed down to a minimum of three.

It is acceptable to start with these three roles held concurrently, but an arrangement where the owner also handles operations should be avoided. When both the business perspective and the technical operations perspective are concentrated in a single person, the balance between KPIs and alerts tends to break down. In practice, this manifests as the business side loosening SLO management in pursuit of results, or overlooking cost overruns by classifying them as "operationally necessary." Separating the roles creates a structure of mutual checks, which leads to more stable operations over the long term.

Among the questions most frequently heard from practitioners when adopting AgentOps, this section addresses one important topic not covered in detail elsewhere in this guide — focusing specifically on the typical failure patterns that emerge during production deployment.

Q. Why does an agent operation that ran smoothly in a pilot break down when deployed to production?

Failure patterns can be broadly organized into three categories.

• Cost failure: Costs estimated from pilot usage levels balloon significantly under production load. Not only does the number of users increase, but the number of reasoning steps per task also tends to grow (because more complex cases get delegated to the agent). Before going to production, always prepare cost projections for expected peak load and set alert thresholds, and design automatic shutdown or HITL fallback mechanisms for when those thresholds are exceeded.
• Audit failure: Audit logs that were not collected during the pilot turn out to be required in production for PDPA compliance or internal controls. Retrofitting log infrastructure means losing historical data, so audit requirements should be anticipated from the start and built into the structured log design. In particular, masking rules for interactions involving PII must be decided in advance.
• HITL failure: Items that were individually approved by administrators during the pilot become too numerous for reviewers to keep up with once volume increases in production. HITL thresholds and sampling rates must be redesigned with production-scale assumptions in mind, and reviewer SLAs and required headcount must be estimated accordingly. Reviewer shortage is the most common cause of HITL failure, so workforce planning should be started early.

The transition from pilot to production should be treated not as "scaling up the same thing," but as a stage of redesigning for production-level load. The human-AI division of labor discussed in the Hybrid BPO Guide also needs to be revisited as scale changes. Concretely, a split that was "80% AI / 20% human" during the pilot often shifts to "60% AI / 40% human" in production. This should be understood not as a quality issue, but as a natural expansion of the domain requiring human intervention, including HITL.

AgentOps is a framework for treating AI agents not as a "tool deployment," but as a formally recognized operational workload within the organization.

The recommended adoption sequence consists of three steps:

• Step 1: Use an agent registry to take inventory of the current state and formally document ownership
• Step 2: Define at least four SLI / SLO and cost metrics, and integrate observability into the existing SRE / DevOps team
• Step 3: Reach agreement with business unit owners on HITL thresholds and escalation paths

Rather than assembling a dedicated team or specialized tooling, clarifying roles and decision-making authority comes first. This was consistently necessary to make production operations work — whether in internal AI assistant adoption, Lao language agent implementation, or elsewhere. AgentOps is better understood as a discussion of operational governance in the AI era than as a purely technical topic — and that framing tends to make it easier to build consensus on the ground.

For related reading, combining the Internal AI Assistant guide, the Hybrid BPO Guide, and the MCP Protocol Introduction will give you a three-dimensional view of how AgentOps connects to your existing operations. As a concrete next step — rather than staying at the level of abstraction — try selecting one of your organization's agents and creating a single inventory sheet based on the five components outlined in this guide.