An Azure video analytics service that uses AI to extract actionable insights from stored videos.
You should start treating the AI agent as an orchestration layer rather than as an autonomous user. Let the agent interpret the request and select an appropriate action, but keep authorization, validation, and execution outside the model. This creates a clear security boundary and limits the impact of incorrect model decisions, prompt injection, or inappropriate tool calls.
Define exactly what actions the agent can perform by exposing a limited set of registered tools or APIs, ideally using well-defined schemas such as OpenAPI. When the agent generates a tool call, pass it through an API or gateway layer that validates the operation and its parameters against defined schemas, data types, and business rules before sending it downstream. The enterprise API should still perform its own authorization rather than trusting the agent. When an action is performed on behalf of a user, preserve that user's identity so existing RBAC and resource-level permissions continue to apply.
Note that not every agent action might need human approval. A risk-based approach is more practical. Read-only and low-risk operations such as retrieving information, generating reports, or drafting content can generally run automatically. Moderate-risk operations can require approval when specific conditions are met, such as a transaction exceeding a defined amount or an operation targeting a sensitive resource. High-risk operations such as financial transactions, deleting customer data, changing security settings, or modifying infrastructure should generate a proposed action and pause until an authorized person approves it. The approval should be enforced by the workflow rather than by the AI agent.
For authentication, use standard enterprise identity mechanisms rather than giving the agent privileged credentials. For actions performed on behalf of a signed-in user, OAuth 2.0 delegated access and the Microsoft Entra On-Behalf-Of flow can preserve the user's identity as the request moves between services. For autonomous background agents that do not represent an interactive user, use Microsoft Entra managed identities or appropriately scoped service principals. If a legacy system requires an API key, keep the key in Azure Key Vault or have an API gateway add it to the request. The agent should never receive the raw secret.
Apply least privilege to individual operations. An agent that can read customer records does not automatically need permission to modify them, and an agent that can create records does not necessarily need permission to delete them. Scope permissions by operation, resource, identity, and environment. The agent can request an action, but the API or workflow layer should remain the final authority on whether that action is allowed.
For monitoring and troubleshooting, give every agent transaction a correlation or trace ID that follows it through the agent, selected tool, authorization decision, downstream APIs, and final state change. Log relevant tool calls, policy decisions, approval events, execution results, errors, latency, and model-related telemetry while avoiding credentials and unnecessary sensitive data. This allows you to determine whether a failure came from the model's interpretation, tool selection, authorization, workflow execution, or a downstream system.
Design write operations for safe retries. Use idempotency keys, transaction identifiers, state tracking, and appropriate retry policies so that a timeout does not cause the agent to create the same record or transaction twice. For multi-step workflows, maintain explicit state and define appropriate recovery or compensation actions when a later step fails.
In a Microsoft environment, you can combine Azure AI Agent Service or Semantic Kernel for agent orchestration and structured tool calls, Entra ID for user and workload identity, Azure API Management as the controlled API boundary, and Azure Logic Apps or Power Automate for deterministic workflows and human approvals. Azure Functions can provide custom APIs or tool implementations, Azure Key Vault can protect credentials required by legacy integrations, and Azure Monitor/Application Insights can provide distributed telemetry.
A practical architecture is: user → AI agent → registered tool → API Management/policy layer → Entra authentication and authorization → optional human approval → deterministic workflow → enterprise system. The agent determines what it thinks should happen; the security and workflow layers independently determine whether that action is permitted and then execute it.
If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.
hth
Marcin