Azure api manger buffers the entire SSE stream before forwarding it to clients when the backend is an Azure OpenAI endpoin

Question

Azure api manger buffers the entire SSE stream before forwarding it to clients when the backend is an Azure OpenAI endpoin

Jeffrey Bakker 0

APIM's internal CacheModelProcessStreamHandlerLegacybuffers the entire SSE stream before forwarding it to clients when the backend is an Azure OpenAI endpoint, despite buffer-request-body="true" buffer-response="false" being explicitly set on forward-request. This defeats streaming and significantly increases TTFT. The "Legacy" suffix suggests a non-buffering implementation exists but is not being used. Even when removing our token limet and llm-emit metric policies we still get this buffering behavior. A month or 2 ago this was not the case.
* User's image

**

Mohammad Altaf 450 Reputation points Microsoft External Staff Moderator

2026-09-02T08:18:44.09+00:00
Hello @Jeffrey Bakker ,

Welcome to Microsoft Q&A,

We understand that Azure API Management is buffering the Server-Sent Events (SSE) response from the Azure OpenAI backend, resulting in the complete response being delivered only after generation is finished, even though buffer-response="false" is configured and the token limit/LLM metric policies have been removed.

To help further isolate the behavior, could you please provide the following details:

APIM SKU

APIM Region

Azure OpenAI Region

A sanitized APIM trace for an affected request

Confirmation whether the same request streams correctly when sent directly to the Azure OpenAI endpoint without APIM in the request path

According to Microsoft documentation, SSE responses should be relayed immediately when buffer-response="false" is configured. Additionally, certain diagnostics or response-processing configurations can impact streaming behavior.

Once we have the above information, we can further analyze where the buffering is occurring and advise on the appropriate resolution.

Thank you!
Mohammad Altaf 450 Reputation points Microsoft External Staff Moderator

2026-09-03T09:29:58.72+00:00

Hello @Jeffrey Bakker

We are following up on our previous request. Could you please share the requested details, including the APIM trace and the result of testing the Azure OpenAI endpoint directly without APIM?

This information will help us further investigate the SSE buffering behavior.

Thank you!
Mohammad Altaf 450 Reputation points Microsoft External Staff Moderator

2026-09-04T10:50:49.0633333+00:00

Hello @Jeffrey Bakker ,

I Just wanted to follow up on my previous comment. When you have a chance, could you please provide the requested details?

Thank you!

1 answer

Your answer

Mohammad Altaf 450 Reputation points Microsoft External Staff Moderator

2026-09-02T08:18:44.09+00:00

Hello @Jeffrey Bakker ,

Welcome to Microsoft Q&A,

We understand that Azure API Management is buffering the Server-Sent Events (SSE) response from the Azure OpenAI backend, resulting in the complete response being delivered only after generation is finished, even though buffer-response="false" is configured and the token limit/LLM metric policies have been removed.

To help further isolate the behavior, could you please provide the following details:

APIM SKU

APIM Region

Azure OpenAI Region

A sanitized APIM trace for an affected request

Confirmation whether the same request streams correctly when sent directly to the Azure OpenAI endpoint without APIM in the request path

According to Microsoft documentation, SSE responses should be relayed immediately when buffer-response="false" is configured. Additionally, certain diagnostics or response-processing configurations can impact streaming behavior.

Once we have the above information, we can further analyze where the buffering is occurring and advise on the appropriate resolution.

Thank you!
Mohammad Altaf 450 Reputation points Microsoft External Staff Moderator

2026-09-03T09:29:58.72+00:00

Hello @Jeffrey Bakker

We are following up on our previous request. Could you please share the requested details, including the APIM trace and the result of testing the Azure OpenAI endpoint directly without APIM?

This information will help us further investigate the SSE buffering behavior.

Thank you!
Mohammad Altaf 450 Reputation points Microsoft External Staff Moderator

2026-09-04T10:50:49.0633333+00:00

Hello @Jeffrey Bakker ,

I Just wanted to follow up on my previous comment. When you have a chance, could you please provide the requested details?

Thank you!

Answer 1

Hi @Jeffrey Bakker

The reported behavior indicates that Azure API Management is buffering the Server-Sent Events (SSE) response from the Azure OpenAI backend instead of forwarding the streamed content to the client as it is generated. For SSE scenarios, the forward-request policy should be configured with buffer-response="false" so that events are relayed immediately to the client. Microsoft documentation also notes that response inspection, validation, caching, and request/response body diagnostics can affect streaming behavior and should be reviewed when troubleshooting SSE-related issues.

Since buffer-response="false" was already configured and the token limit and LLM metric policies were removed, the next step is to verify whether the streaming behavior is observed only through API Management or also when calling the Azure OpenAI endpoint directly. Additionally, review any diagnostics or response-processing configurations that may be inspecting or logging response bodies, as these can impact streaming behavior.

If the issue persists, please reopen the thread and provide the APIM SKU, APIM region, Azure OpenAI region, and an APIM trace for a sample request so that further analysis can be performed.

If the assistance was helpful, kindly take a moment to click on Accept Answer and click on Yes. It will be helpful for other community members.

Thank you.