For years, companies have been asking themselves how to govern AI-generated responses. But what happens when artificial intelligence starts to act?
Investments in Agentic AI solutions continue to be substantial. According to EY’s AI Pulse Survey, approximately one-third of companies expect to invest more than $10 million in artificial intelligence in 2026, to give agents an increasingly central role in executing business processes.
The enthusiasm is understandable: many see AI as the future operating system of the enterprise, but this also brings new responsibilities. When a system not only generates content but also makes decisions, interacts with other systems, and performs actions that impact the business, governance becomes an absolute priority.
Key Points
- The agentic paradigm introduces new risks compared with generative AI because of its ability to make decisions and execute actions.
- Building an Agentic AI governance model means addressing compliance, security, accountability, and oversight across the entire ecosystem.
- Governance must accompany the agents’ lifecycle, from design through continuous monitoring and, eventually, decommissioning.
Why Agentic Systems Change the Rules of AI Governance
For years, AI governance has focused on model outputs: given a prompt and a response, it was necessary to verify accuracy, transparency, bias, and regulatory compliance. The final decision remained with a professional, who would act based on their own experience and the AI model’s output.
With agentic systems, the paradigm changes because these software systems:
- plan activities;
- call tools;
- use credentials;
- access data;
- use APIs and corporate services;
- collaborate with other agents within workflows;
- execute sequences of actions to achieve a goal.
Complicating the picture further is the issue of accountability. When an agent makes decisions and executes actions, who is responsible? The accountability chain becomes longer, involving not only those who developed the model, but also those who integrated and configured the solution and, above all, the company that decides to entrust it with specific activities.
AI governance in an agentic context therefore cannot be limited to verifying the reliability of the model on which solutions are based. It must also monitor in real time what the agent is authorized to do, which resources it can use and when it must stop, while also establishing robust accountability models.
The New Risks of the Agentic Era
Autonomy and the ability to act represent the main points of discontinuity between agentic systems and generative AI. Managing them effectively requires considering new categories of risk, including the agent’s interaction with the external ecosystem.
- Loss of control over task execution
An agent can chain together dozens of actions. If objectives and constraints are not defined correctly, errors can propagate and become extremely difficult to detect. - Excessive permissions and authorizations
Agents access APIs and databases to perform their tasks. It is essential that the privileges granted never exceed what is strictly necessary. - New cyber risks
The more an agent is connected to systems, data and applications, the greater the number of points an attacker can attempt to exploit for their own benefit. - Unpredictable behavior in multi-agent systems
Interactions between agents, even when governed by deterministic logic, can produce unexpected effects. It is necessary to define not only rules, but also monitoring and arbitration mechanisms capable of detecting unforeseen effects in real time. - Difficulty in auditing and establishing accountability
Reconstructing who made a decision, which information they used and why they executed a particular action can be extremely complex within ecosystems that orchestrate hundreds of different software agents, often developed by different vendors and interconnected with one another.
Shared Responsibility: The Three Levels of Accountability
Accountability in agentic ecosystems operates across three distinct levels: technology, corporate governance, and individual operational roles.
- Technological ecosystem
Each actor is responsible for their own area of responsibility. The model provider is accountable for its capabilities and technological limitations; those who develop and integrate the solution are responsible for the architecture and control mechanisms implemented at the company’s direction; the company itself retains ultimate responsibility for how the solution is used, deciding its objectives, level of autonomy and the responsibilities of the professionals involved. - Corporate organization
At the corporate level, clear governance is essential. In practical terms, it must be clear who defines policies, approves use cases, assigns authorizations and access levels and decides when an activity requires human intervention.
These decisions should be entrusted to a cross-functional governance body (risk, compliance, security and business), rather than to a single function such as IT, in order to balance operational needs with legal and reputational risks. - Clearly defined roles and responsibilities
A fundamental part of governance is defining who monitors the agent’s behavior and periodically reassesses its permissions and levels of autonomy, who ensures the traceability of its actions, and who has the authority to suspend it in the event of anomalous behavior. Finally, it is necessary to establish who trains professionals to recognize the agent’s limitations.
Not Just Compliance: Agent Governance Goes Beyond Regulatory Requirements
It may be tempting to think that investing in AI governance is solely about ensuring compliance. To some extent, this is true: the European AI Act, to cite a current example, introduces stringent obligations for AI systems, particularly high-risk systems, including human oversight, traceability and risk management throughout the entire lifecycle.
This is complemented, for example, by the requirements of the ISO 42001 standard, which provides a structured framework for policies, risk assessment and continuous improvement, applicable beyond European borders as well.
Reducing governance to a compliance exercise would, however, be a mistake in perspective. An agent operating with excessive permissions and without adequate controls exposes the company to risks that go beyond sanctions and include concrete operational damage, security incidents and reputational repercussions. Regulatory compliance, in this sense, is the minimum threshold, not the goal.
Governance Covers the Entire Agent Lifecycle
A common mistake is to think of agentic system governance as a set of controls and processes to be applied once the system is already ready for production.
- The critical moment for preventing risks comes much earlier, particularly during the design phase, when objectives, constraints, the data the agent will be able to access, and the level of autonomy granted are defined.
- During development and testing, governance translates into security checks, simulations of adverse scenarios, and validation of control mechanisms before the agent interacts with real systems or data.
- During deployment, authorizations, minimum necessary permissions, and thresholds for human intervention come into play.
- In day-to-day operations, governance must be supported by continuous monitoring, periodic audits, the reassessment of permissions, incident management, and the ability to detect and correct deviations from the original objectives.
- Finally, decommissioning also requires attention: it is necessary to properly revoke credentials and access, verify that no business process depends on the decommissioned agent, and retain the documentation needed for future audits.
Designing governance as a continuous cycle, rather than as an initial control, is what distinguishes mature agent adoption from a risky implementation.
Kirey: Innovating with AI Without Losing Control
Digital transformation requires not only new technologies, but also a new way of governing them. When AI ceases to be a simple support tool and becomes an integral part of business processes, companies need a partner capable of supporting them in defining an effective governance model. Without one, they risk facing compliance issues, new vulnerabilities, undefined responsibilities, and the loss of control over automated processes.
At Kirey, we help companies transform artificial intelligence from a collection of experiments and isolated initiatives into the next operating system for the entire organization. We stand alongside them to unlock its full potential, without losing sight of key issues such as control, security, compliance, and risk management. The design and implementation of governance models for agentic AI are precisely aimed at this goal.
Contact us to begin the journey toward more autonomous, secure, and well-governed AI together.
