PlayFab 與 Azure 間經常性逾時

Question

PlayFab 與 Azure 間經常性逾時

Daniel 20

在過去十天中，我們的 Unity 用戶端使用 PlayFab API 所發出的請求經常會發生逾時，並接收到 PlayFab 的 CloudScriptAzureFunctionsExecutionTimeLimitExceeded (1470) 的錯誤碼。
前往 Azure 查看發生逾時的請求函式的引動過程，則會顯示錯誤，並且僅顯示 An operation was attempted on a nonexistent network connection. (0x800704CD) 的錯誤訊息，並且連函式第一行的 Log 都沒有執行。且有發現並不是每次逾時錯誤的請求皆會被記錄到引動過程中。

發生逾時的請求不論執行函式的大小，皆有收到過逾時的錯誤碼。但是過幾秒鐘後再次發送請求則又會正常，屬於機率性發生的狀況。

如果能提供偵錯方向的話，感激不盡

匿名

2025-05-15T06:26:26.0933333+00:00
Hi Daniel,

As per the error message "An operation was attempted on a nonexistent network connection. (0x800704CD)" it looks like a network issue.

To debug this issue, consider the following steps:

Ensure that the Azure Function's environment is configured correctly and meets the required specifications.

Check the execution time limit for your Azure Function and see if it needs to be adjusted.

Investigate any network connectivity issues that might be affecting the invocation.

Monitor the Azure Function's performance to identify any patterns or specific conditions leading to the timeouts.

Additionally, you may want to review the Azure logs for more detailed error messages that could provide further insight.

In case if you have validated this and still facing the issue then I will suggest you review the Diagnose and solve problems blade on your Function app to validate if there are any insights that provide more details on your error/exception/issue and mitigation steps to resolve the issue.

To help you better understand, kindly refer to the documentations below:

https://stackoverflow.com/questions/30706400/an-operation-was-attempted-on-a-nonexistent-network-connection-error-code-1229

By following the steps above, you can better understand the root cause of the timeout issue. Hope these suggestions help with your debugging efforts.

For further troubleshooting, could you please provide the error screenshot? It would be helpful.
Daniel 20 信譽點數

2025-05-16T03:19:35.18+00:00

Hello, thanks for responding!

I checked the Diagnose and solve problems tab but I didn't get anything concrete

We also found out that even when clicking on the Browse button

Sometimes it would take long to load the webpage. PlayFab timeout should be that it encounters one of these long loading times.
匿名

2025-05-19T16:28:29.8366667+00:00
嗨，丹尼尔，

如果使用消耗计划或动态缩放，Azure Functions 在冷启动期间可能会遇到延迟。

如果请求在函数实例完全初始化之前终止，则冷启动可能会导致延迟，并且不会进行日志记录。考虑迁移到 Premium 计划以保持实例温暖。

该错误通常表明 PlayFab 和 Functionapp 之间存在网络套接字问题。

如果 PlayFab 由于暂时性网络问题或服务中断而无法连接，它可能会返回超时或无提示失败。

由于限制、DNS 解析问题或服务内部配置错误，PlayFab 可能会以静默方式无法调用 Azure Functions。

使用 Application Insights 更有效地记录和关联 Azure Functions 中的故障。

检查 Azure 指标，了解函数超时、失败执行和限制。

检查 Azure 函数的 VNET 中是否存在任何出站网络限制、服务终结点或 DNS 问题。
匿名

2025-05-20T18:13:33.1633333+00:00

嗨，丹尼尔，只是查看以上提供的信息是否有帮助。如果您有任何问题，请告诉我们。
Daniel 20 信譽點數

2025-05-21T02:06:03.3633333+00:00

Hello, @Anonymous Thank you for responding!

We don't think it's cold start, although we do observe there's cold start delay using Azure's Diagnose and Solve Problems/Availability and Performance/Function Cold Start. The timeout doesn't just happen at the first API call. It would happen to the second, third or twentieth of consecutive API call. (We test this with our in-game buttons manually)

Also we created a new Function App with the same code and environment variables, the timeout seems to disappear. The same PlayFab server connection is used, too.

We now suspect it's about durable functions and task hub, I will report back today when we've done the adjustment and some testing. If the problem persists, we could switch to the new function app and hopefully the problem wouldn't carry to that one.

Thank you!
匿名

2025-05-21T03:25:52.0366667+00:00

当然，Daniel，感谢您的更新。
Daniel 20 信譽點數

2025-05-21T08:30:20.38+00:00

I'm sorry to report that it's not about task hub, so our original function app would still encounter timeout.

Pretty sure it's about Azure not responding, because we have multiple devices that would encounter long loading time at the same time. So if one device is waiting, every request at that moment would have to wait till Azure starts responding again in 10 to sometimes 20 seconds later. At that moment if I open our function app's website (screenshot above), browser would join the wait.

So the problem still exists in our old function app.
匿名

2025-05-22T11:52:46.0233333+00:00

Daniel，您是否在 Application Insights 日志中观察到任何错误？

在 Azure 门户中使用 App Insights 或诊断和解决问题 => 网络 => SNAT 端口耗尽，并检查是否看到 SocketException、 connect ECONNRESET或 connection refused 日志。
已刪除

由於違反我們的規範，因此已刪除此註解。在採取動作之前，會透過自動化偵測手動報告或識別註解。如需詳細資訊，請參閱我們的管理辦法。
已刪除

由於違反我們的規範，因此已刪除此註解。在採取動作之前，會透過自動化偵測手動報告或識別註解。如需詳細資訊，請參閱我們的管理辦法。
Daniel 20 信譽點數

2025-05-23T05:41:38.3966667+00:00

Hello @Anonymous

We have multiple Function Apps that for different purposes. Right now it’s the one for development that has the problem, the other ones don’t have the same problem, but they also don’t have all the configurations and codes the development server has right now. The development Function App is the newest.

When the API times out on the PlayFab side, Azure function invocation would only show “an operation was attempted on a nonexistent network connection”, it won’t show even the log on the first line of the function.

Sometimes timeout API calls don’t get recorded in the invocation tab. We also found there’s COM Exceptions and sometimes Socket Exceptions every day since the problem started, through Diagnose and Solve Problems. I think it’s why there’s no recorded.

We tested with local backend, bypass the Azure Function App, and there’s no timeout happening so far.

There’s no SNAT port exhaustion data on Azure, but I looked up the Internet and found that it usually doesn’t have to setup NAT gateway for most PlayFab related situations.

There’s still cold start, despite us pinging Azure every 4 minutes. But when we keep sending APIs, like changing in-game character’s skin every a couple of seconds with manual operations, there would still be timeout after several tries, so I don’t think it’s all about cold start.

I think it’s most likely our fault for causing the timeout, maybe some configurations, but I couldn’t figure out where the problem is.

We now switch to a newly created Function App with the same code and the same environment variables. There's no unreasonable timeout occurred so far.

Our Function and Error tab

Our SNAT port exhaustion tab

您的回答

匿名

2025-05-15T06:26:26.0933333+00:00

Hi Daniel,

As per the error message "An operation was attempted on a nonexistent network connection. (0x800704CD)" it looks like a network issue.

To debug this issue, consider the following steps:

Ensure that the Azure Function's environment is configured correctly and meets the required specifications.

Check the execution time limit for your Azure Function and see if it needs to be adjusted.

Investigate any network connectivity issues that might be affecting the invocation.

Monitor the Azure Function's performance to identify any patterns or specific conditions leading to the timeouts.

Additionally, you may want to review the Azure logs for more detailed error messages that could provide further insight.

In case if you have validated this and still facing the issue then I will suggest you review the Diagnose and solve problems blade on your Function app to validate if there are any insights that provide more details on your error/exception/issue and mitigation steps to resolve the issue.

To help you better understand, kindly refer to the documentations below:

https://stackoverflow.com/questions/30706400/an-operation-was-attempted-on-a-nonexistent-network-connection-error-code-1229

By following the steps above, you can better understand the root cause of the timeout issue. Hope these suggestions help with your debugging efforts.

For further troubleshooting, could you please provide the error screenshot? It would be helpful.
Daniel 20 信譽點數

2025-05-16T03:19:35.18+00:00

Hello, thanks for responding!

I checked the Diagnose and solve problems tab but I didn't get anything concrete

We also found out that even when clicking on the Browse button

Sometimes it would take long to load the webpage. PlayFab timeout should be that it encounters one of these long loading times.
匿名

2025-05-19T16:28:29.8366667+00:00

嗨，丹尼尔，

如果使用消耗计划或动态缩放，Azure Functions 在冷启动期间可能会遇到延迟。

如果请求在函数实例完全初始化之前终止，则冷启动可能会导致延迟，并且不会进行日志记录。考虑迁移到 Premium 计划以保持实例温暖。

该错误通常表明 PlayFab 和 Functionapp 之间存在网络套接字问题。

如果 PlayFab 由于暂时性网络问题或服务中断而无法连接，它可能会返回超时或无提示失败。

由于限制、DNS 解析问题或服务内部配置错误，PlayFab 可能会以静默方式无法调用 Azure Functions。

使用 Application Insights 更有效地记录和关联 Azure Functions 中的故障。

检查 Azure 指标，了解函数超时、失败执行和限制。

检查 Azure 函数的 VNET 中是否存在任何出站网络限制、服务终结点或 DNS 问题。
匿名

2025-05-20T18:13:33.1633333+00:00

嗨，丹尼尔，只是查看以上提供的信息是否有帮助。如果您有任何问题，请告诉我们。
Daniel 20 信譽點數

2025-05-21T02:06:03.3633333+00:00

Hello, @Anonymous Thank you for responding!

We don't think it's cold start, although we do observe there's cold start delay using Azure's Diagnose and Solve Problems/Availability and Performance/Function Cold Start. The timeout doesn't just happen at the first API call. It would happen to the second, third or twentieth of consecutive API call. (We test this with our in-game buttons manually)

Also we created a new Function App with the same code and environment variables, the timeout seems to disappear. The same PlayFab server connection is used, too.

We now suspect it's about durable functions and task hub, I will report back today when we've done the adjustment and some testing. If the problem persists, we could switch to the new function app and hopefully the problem wouldn't carry to that one.

Thank you!
匿名

2025-05-21T03:25:52.0366667+00:00

当然，Daniel，感谢您的更新。
Daniel 20 信譽點數

2025-05-21T08:30:20.38+00:00

I'm sorry to report that it's not about task hub, so our original function app would still encounter timeout.

Pretty sure it's about Azure not responding, because we have multiple devices that would encounter long loading time at the same time. So if one device is waiting, every request at that moment would have to wait till Azure starts responding again in 10 to sometimes 20 seconds later. At that moment if I open our function app's website (screenshot above), browser would join the wait.

So the problem still exists in our old function app.
匿名

2025-05-22T11:52:46.0233333+00:00

Daniel，您是否在 Application Insights 日志中观察到任何错误？

在 Azure 门户中使用 App Insights 或诊断和解决问题 => 网络 => SNAT 端口耗尽，并检查是否看到 SocketException、 connect ECONNRESET或 connection refused 日志。
已刪除

由於違反我們的規範，因此已刪除此註解。在採取動作之前，會透過自動化偵測手動報告或識別註解。如需詳細資訊，請參閱我們的管理辦法。
Daniel 20 信譽點數

2025-05-23T05:41:38.3966667+00:00

Hello @Anonymous

We have multiple Function Apps that for different purposes. Right now it’s the one for development that has the problem, the other ones don’t have the same problem, but they also don’t have all the configurations and codes the development server has right now. The development Function App is the newest.

When the API times out on the PlayFab side, Azure function invocation would only show “an operation was attempted on a nonexistent network connection”, it won’t show even the log on the first line of the function.

Sometimes timeout API calls don’t get recorded in the invocation tab. We also found there’s COM Exceptions and sometimes Socket Exceptions every day since the problem started, through Diagnose and Solve Problems. I think it’s why there’s no recorded.

We tested with local backend, bypass the Azure Function App, and there’s no timeout happening so far.

There’s no SNAT port exhaustion data on Azure, but I looked up the Internet and found that it usually doesn’t have to setup NAT gateway for most PlayFab related situations.

There’s still cold start, despite us pinging Azure every 4 minutes. But when we keep sending APIs, like changing in-game character’s skin every a couple of seconds with manual operations, there would still be timeout after several tries, so I don’t think it’s all about cold start.

I think it’s most likely our fault for causing the timeout, maybe some configurations, but I couldn’t figure out where the problem is.

We now switch to a newly created Function App with the same code and the same environment variables. There's no unreasonable timeout occurred so far.

Our Function and Error tab

Our SNAT port exhaustion tab