429s through API Management in front of Azure OpenAI, but not when calling the deployment directly

Question

429s through API Management in front of Azure OpenAI, but not when calling the deployment directly

Aria Rahujani 20

I’m running a batch job that sends sequential chat-completion requests to a gpt-5.1 deployment, roughly 2 requests per minute at about 4,700 prompt tokens and about 110 completion tokens each.

Calling the Azure OpenAI endpoint directly: no 429s.
Calling the same workload through an Azure API Management gateway (API imported for Azure OpenAI, attached to a product, subscription key required): nearly every request returns 429 on the first attempt and succeeds after the Retry-After delay.
Smaller requests (about 1,600 tokens each, 3 per minute) through the same gateway return no 429s.

What I’ve checked so far:

The API’s own policy has no llm-token-limit, azure-openai-token-limit, rate-limit or rate-limit-by-key element.

Questions:

Besides API-level policies, where else can a token or rate limit be applied (product, global, policy fragments, something configured through the AI Foundry gateway integration)? Is there a way to see the effective policy for one operation?
How can I tell whether a 429 was returned by API Management itself or passed through from the Azure OpenAI backend? Application Insights is enabled on the API.
If the API Management backend points at the same deployment as my direct endpoint, what could make the gateway’s effective throughput lower?I’m running a batch job that sends sequential chat-completion requests to a gpt-5.1 deployment, roughly 2 requests per minute at about 4,700 prompt tokens and about 110 completion tokens each.
- Calling the Azure OpenAI endpoint directly: no 429s.
- Calling the same workload through an Azure API Management gateway (API imported for Azure OpenAI, attached to a product, subscription key required): nearly every request returns 429 on the first attempt and succeeds after the Retry-After delay.
- Smaller requests (about 1,600 tokens each, 3 per minute) through the same gateway return no 429s.
What I’ve checked so far:
- The API’s own policy has no llm-token-limit, azure-openai-token-limit, rate-limit or rate-limit-by-key element.
Questions:
1. Besides API-level policies, where else can a token or rate limit be applied (product, global, policy fragments, something configured through the AI Foundry gateway integration)? Is there a way to see the effective policy for one operation?
2. How can I tell whether a 429 was returned by API Management itself or passed through from the Azure OpenAI backend? Application Insights is enabled on the API.
3. If the API Management backend points at the same deployment as my direct endpoint, what could make the gateway’s effective throughput lower?

REDDIMALLA VENKATESH 0 Reputation points Microsoft External Staff

2026-10-22T11:31:23.6833333+00:00
Hello @Aria Rahujani ,

Thank you for providing the detailed information and for sharing the comparison between direct Azure OpenAI requests and requests routed through Azure API Management.

I understand that the direct calls to the same deployment are completing successfully, while the larger requests routed through API Management are receiving HTTP 429 responses on the initial attempt and succeeding after the Retry-After interval. I also noted that smaller requests through the same gateway are not experiencing the issue.

To help isolate whether the 429 response is being generated at the API Management layer or returned by the Azure OpenAI backend, could you please share the following from an affected request:

Approximate UTC timestamp of the request

APIM request/correlation ID, if available

Complete 429 response headers and error response, with any sensitive information removed

The relevant APIM policy configuration, including inherited/global/product-level policies or policy fragments applicable to this API

Azure OpenAI deployment TPM configuration

Since Application Insights is already enabled, the telemetry from an affected request should also help us correlate the gateway request with the backend call.

Please ensure that subscription keys, API keys, access tokens, or other credentials are removed before sharing the information.

Thank you for your cooperation. We’ll use these details to narrow down where the throttling is occurring and determine the appropriate next steps.

Your answer

REDDIMALLA VENKATESH 0 Reputation points Microsoft External Staff

2026-10-22T11:31:23.6833333+00:00

Hello @Aria Rahujani ,

Thank you for providing the detailed information and for sharing the comparison between direct Azure OpenAI requests and requests routed through Azure API Management.

I understand that the direct calls to the same deployment are completing successfully, while the larger requests routed through API Management are receiving HTTP 429 responses on the initial attempt and succeeding after the Retry-After interval. I also noted that smaller requests through the same gateway are not experiencing the issue.

To help isolate whether the 429 response is being generated at the API Management layer or returned by the Azure OpenAI backend, could you please share the following from an affected request:

Approximate UTC timestamp of the request

APIM request/correlation ID, if available

Complete 429 response headers and error response, with any sensitive information removed

The relevant APIM policy configuration, including inherited/global/product-level policies or policy fragments applicable to this API

Azure OpenAI deployment TPM configuration

Since Application Insights is already enabled, the telemetry from an affected request should also help us correlate the gateway request with the backend call.

Please ensure that subscription keys, API keys, access tokens, or other credentials are removed before sharing the information.

Thank you for your cooperation. We’ll use these details to narrow down where the throttling is occurring and determine the appropriate next steps.