Microsoft Foundry Hosted Agent: Intermittent 401 PermissionDenied on project chat/completions endpoint despite Foundry User role

Marzieh Fattahi 0 Reputation points
2026-11-01T16:52:29.5933333+00:00

Setup

I am running a Microsoft Foundry Hosted Agent using a custom Python container and the Invocations protocol. The agent calls a GPT-4.1-mini deployment through the Foundry project endpoint:

POST https://<account>.services.ai.azure.com/api/projects/<project>/openai/v1/chat/completions

The OpenAI client is obtained through AIProjectClient.get_openai_client().

Authentication uses DefaultAzureCredential, which resolves to ManagedIdentityCredential inside the hosted container, with the scope https://ai.azure.com/.default.

The dependencies are pinned and identical across all runs:

azure-identity 1.26.0

azure-ai-projects 2.8.0

openai 3.31.0

langchain-openai 1.7.2

azure-ai-agentserver-core 2.2.0

azure-ai-agentserver-invocations 1.2.0

SDK retries are disabled.

Problem: Intermittent HTTP 401 PermissionDenied

I performed three diagnostic runs:

1. Version A – First run (failed)

The first chat/completions request returned HTTP 401 with error code PermissionDenied.

At that time, the Hosted Agent identity only had the implicit project access described in Microsoft's documentation.

2. Version A – Second run (successful)

I assigned the built-in Foundry User role to the agent identity at project scope.

Approximately 30 minutes later, the same Hosted Agent version and container image successfully completed four GPT-4.1-mini requests.

3. Version B – Third run (failed)

Approximately 16 hours later, I created a new Hosted Agent version with application-code changes.

The following remained unchanged:

Agent identity and project-scoped role assignments

Model authentication code

SDK versions and dependencies

Token scope and project endpoint

Environment variables

Nevertheless, the first chat/completions request again returned HTTP 401 PermissionDenied.

What I have verified

Both authenticated startup calls to my backend API succeeded in the third run. The backend recorded the expected agent identity as the caller.

The explicit project-scoped Foundry User role assignment is still present, unconditional and unchanged.

The project's managed identity still has Foundry User at the Foundry account scope.

Azure Activity Log shows no relevant RBAC, identity, account, project or model-deployment changes between the successful and failed runs.

Foundry account metrics recorded four successful model requests during the second run, but zero model requests during the two failed runs.

The failed runs also show zero client errors in account-level metrics, suggesting the requests were rejected before reaching the model deployment.

The model authentication implementation and installed SDK dependencies are identical between Versions A and B.

Every run was a cold start. In the third run, the managed identity token was acquired milliseconds before the rejected request.

I have not captured the claims of the actual ai.azure.com token rejected by the model endpoint, and I cannot access the gateway-side authorization decision.

Therefore, I cannot yet determine why the request was rejected.

Questions

Does /api/projects/<project>/openai/v1/chat/completions require permissions beyond the documented implicit Hosted Agent project access or an explicit project-scoped Foundry User role? Does the documented responses/* permission also cover chat/completions?

Can creating a new Hosted Agent version or session change the effective identity or authorization context used for outbound model requests, even when the configured agent identity and permissions remain unchanged?

Could gateway-side authorization caching, or re-evaluation when a new version is created, explain why a project-scoped assignment that worked for Version A is not honoured for Version B?

What diagnostics would you recommend to identify the rejected token's principal and audience, and the gateway's authorization decision, without exposing access tokens or assigning broader permissions? Is there a diagnostic-log category or another supported method for investigating denied project-endpoint requests?

I can share the relevant UTC timestamps and provider request IDs privately with a Microsoft engineer.

I would appreciate guidance on identifying which component returns PermissionDenied and how to resolve the issue without unnecessarily broadening RBAC permissions.

Setup

I am running a Microsoft Foundry Hosted Agent using a custom Python container and the Invocations protocol. The agent calls a GPT-4.1-mini deployment through the Foundry project endpoint:

POST https://<account>.services.ai.azure.com/api/projects/<project>/openai/v1/chat/completions

The OpenAI client is obtained through AIProjectClient.get_openai_client().

Authentication uses DefaultAzureCredential, which resolves to ManagedIdentityCredential inside the hosted container, with the scope https://ai.azure.com/.default.

The dependencies are pinned and identical across all runs:

azure-identity 1.26.0

azure-ai-projects 2.8.0

openai 3.31.0

langchain-openai 1.7.2

azure-ai-agentserver-core 2.2.0

azure-ai-agentserver-invocations 1.2.0

SDK retries are disabled.

Problem: Intermittent HTTP 401 PermissionDenied

I performed three diagnostic runs:

1. Version A – First run (failed)

The first chat/completions request returned HTTP 401 with error code PermissionDenied.

At that time, the Hosted Agent identity only had the implicit project access described in Microsoft's documentation.

2. Version A – Second run (successful)

I assigned the built-in Foundry User role to the agent identity at project scope.

Approximately 30 minutes later, the same Hosted Agent version and container image successfully completed four GPT-4.1-mini requests.

3. Version B – Third run (failed)

Approximately 16 hours later, I created a new Hosted Agent version with application-code changes.

The following remained unchanged:

Agent identity and project-scoped role assignments

Model authentication code

SDK versions and dependencies

Token scope and project endpoint

Environment variables

Nevertheless, the first chat/completions request again returned HTTP 401 PermissionDenied.

What I have verified

Both authenticated startup calls to my backend API succeeded in the third run. The backend recorded the expected agent identity as the caller.

The explicit project-scoped Foundry User role assignment is still present, unconditional and unchanged.

The project's managed identity still has Foundry User at the Foundry account scope.

Azure Activity Log shows no relevant RBAC, identity, account, project or model-deployment changes between the successful and failed runs.

Foundry account metrics recorded four successful model requests during the second run, but zero model requests during the two failed runs.

The failed runs also show zero client errors in account-level metrics, suggesting the requests were rejected before reaching the model deployment.

The model authentication implementation and installed SDK dependencies are identical between Versions A and B.

Every run was a cold start. In the third run, the managed identity token was acquired milliseconds before the rejected request.

I have not captured the claims of the actual ai.azure.com token rejected by the model endpoint, and I cannot access the gateway-side authorization decision.

Therefore, I cannot yet determine why the request was rejected.

Questions

Does /api/projects/<project>/openai/v1/chat/completions require permissions beyond the documented implicit Hosted Agent project access or an explicit project-scoped Foundry User role? Does the documented responses/* permission also cover chat/completions?

Can creating a new Hosted Agent version or session change the effective identity or authorization context used for outbound model requests, even when the configured agent identity and permissions remain unchanged?

Could gateway-side authorization caching, or re-evaluation when a new version is created, explain why a project-scoped assignment that worked for Version A is not honoured for Version B?

What diagnostics would you recommend to identify the rejected token's principal and audience, and the gateway's authorization decision, without exposing access tokens or assigning broader permissions? Is there a diagnostic-log category or another supported method for investigating denied project-endpoint requests?

I can share the relevant UTC timestamps and provider request IDs privately with a Microsoft engineer.

I would appreciate guidance on identifying which component returns PermissionDenied and how to resolve the issue without unnecessarily broadening RBAC permissions.

Microsoft Foundry
Microsoft Foundry

A unified Azure platform for creating and managing AI models, agents, and applications with built‑in enterprise security, monitoring, and governance

0 comments

1 answer

Sort by: Oldest
Most helpful Newest Oldest
  1. Hemant Bharadwaj 355 Reputation points Microsoft Employee
    2026-11-02T16:01:11.8566667+00:00

    Likely cause/summary: this pattern (same code/deps, intermittent 401, zero model requests in metrics during failures) usually means authorization is failing before model execution, often due to role/scope mismatch at data-plane, token audience mismatch, or transient RBAC/token propagation state.

    1. Verify token audience and principal in each failing run.
      • Capture JWT aud, tid, oid/appid from the access token used for chat/completions.
      • Ensure the token audience matches what the Foundry project endpoint expects for that call path.
      • Compare a successful run token vs failed run token.
    2. Confirm effective permissions at all required scopes.
    3. Eliminate stale credentials between agent versions.
      • Force a fresh token acquisition at startup (before first inference call).
      • Log token expiration/issued times and first-call timestamp.
      • Intermittent first-call 401 after version rollout often points to stale cached credential state.
    4. Validate endpoint consistency.
      • Ensure failing and successful runs hit the exact same project endpoint, deployment name, and API path/version.
      • Log resolved endpoint and deployment metadata at startup.
    5. Escalate with a correlation bundle (likely needed if issue persists).

    Product reference:

    Confirmed facts from your post:

    • Same dependencies and identity setup can succeed, then later fail with 401.
    • Failed windows show zero model requests recorded, implying pre-model authorization rejection.

    Possible causes (not yet confirmed):

    • intermittent token audience/scope mismatch,
    • effective-permission gap at one required scope,
    • transient RBAC/token propagation/cache inconsistency at runtime.

    Validation checks:

    • first call succeeds consistently after fresh token acquisition,
    • token claims are identical across good/bad runs,
    • no further 401 under unchanged role assignments.

    “If you find this answer helpful, please upvote it. If this resolves your issue, please mark it as the accepted answer so that it can help others in the community.”

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.