Intermittent HTTP 500 Internal Server Errors on ARM Deployments (germanywestcentral)

Question

Intermittent HTTP 500 Internal Server Errors on ARM Deployments (germanywestcentral)

Laurens Bremers 25

We experienced intermittent Internal Server Errors when executing deployments via Azure Resource Manager on Feb 26 (09:30-16:00 UTC). This issue persisted for several hours with retries resulting in a mix of successes and failures, after which the issue disappeared. We noticed these errors not only in our own Azure Tenant, but also in tenants of our customers that use our deployment scripts to automate their Azure Virtual Desktop deployments (through the Azure Go SDK).

We had a very similar issue on December 18th 2025: intermittent Internal Server Errors during deployments for a few hours, issue being resolved around 16:00 UTC.

During this windows, the Azure Status page reported all services as healthy in this region. Because we do not have a paid support plan, we are unable to open a technical support ticket through the portal to get this platform issue investigated.

Would it be possible for a one-time support ticket to be opened so the root cause can be identified? We suspect a backend node failure or capacity issue in the region.

Some of the event log entries of failed deployments:<REDACTED RESOURCE DATA> <moved to Private Message>

Anonymous

2026-03-10T17:41:03.8133333+00:00
Hello Laurens,

Could you let us know if this issue occurs frequently and across all regions for the same deployment?

Additionally, could you confirm if any failures were reported when manual deployments were performed on the resource from the Azure Portal?
Laurens Bremers 25 Reputation points

2026-03-11T16:01:40.63+00:00

Hi Ankit,

It has happened for us on two occasions (18 dec 2025 and 26 feb 2026). In both cases it was for almost the entire working day (starting 9:30, returning to succeeding deployments around 16:00 UTC).

During these periods, the vast majority of the deployments we started (1-5k), failed with one of the resources in the ARM template reporting an InternalServerError. We no longer have the data for December, but for Februari I did attach some examples.

In both instances, it happened for all the customers that used the deployment script, although it could be limited to the GermanyWestCentral region, as that is most commonly used by our customers in those timezones.

In December, an Azure support engineer of a customer did conclude that normal VM deployments through the portal were also failing. In Februari, a support engineer for the same customer did report being able to deploy 4 VMs through the portal successfully and concluded it had to be a problem in the script. However, this ignores the fact that we also see a portion of the deployments through our script succeed. We believe those 4 successful portal deployments were a coin-flip.

With regards to retries and back-offs. We do properly implement retries and back-offs. However, in this case, the InternalServerErrors were persistent for multiple hours on end.

We also have a mechanism that deletes VMs that fail to deploy and then replaces them with a new deployment. However, in these cases, the re-deployment failed as well. In that case, there would be 5min+ between the first and the second deployment.

Answer accepted by question author

0 additional answers

Your answer

Anonymous

2026-03-10T17:41:03.8133333+00:00

Hello Laurens,

Could you let us know if this issue occurs frequently and across all regions for the same deployment?

Additionally, could you confirm if any failures were reported when manual deployments were performed on the resource from the Azure Portal?
Laurens Bremers 25 Reputation points

2026-03-11T16:01:40.63+00:00

Hi Ankit,

It has happened for us on two occasions (18 dec 2025 and 26 feb 2026). In both cases it was for almost the entire working day (starting 9:30, returning to succeeding deployments around 16:00 UTC).

During these periods, the vast majority of the deployments we started (1-5k), failed with one of the resources in the ARM template reporting an InternalServerError. We no longer have the data for December, but for Februari I did attach some examples.

In both instances, it happened for all the customers that used the deployment script, although it could be limited to the GermanyWestCentral region, as that is most commonly used by our customers in those timezones.

In December, an Azure support engineer of a customer did conclude that normal VM deployments through the portal were also failing. In Februari, a support engineer for the same customer did report being able to deploy 4 VMs through the portal successfully and concluded it had to be a problem in the script. However, this ignores the fact that we also see a portion of the deployments through our script succeed. We believe those 4 successful portal deployments were a coin-flip.

With regards to retries and back-offs. We do properly implement retries and back-offs. However, in this case, the InternalServerErrors were persistent for multiple hours on end.

We also have a mechanism that deletes VMs that fail to deploy and then replaces them with a new deployment. However, in these cases, the re-deployment failed as well. In that case, there would be 5min+ between the first and the second deployment.

Answer 1

Anonymous

Hello Laurens,

We appreciate your patience as we worked with the backend team to investigate the root cause of these deployment failures.

The investigation revealed that on 2026-03-17, there was a transient fault in one backend instance of Azure Resource Manager (ARM) caused failures for any CRUD operations routed to it, while other instances remained unaffected. The issue was detected and resolved automatically after the instance restarted. No configuration changes or deployments triggered the problem, and it was confirmed to be an isolated reliability fault.

To prevent recurrence, the platform team has filed repair items and is working on a fix, aiming for global deployment by May 2026. We apologize for any inconvenience caused.

Recommendations:

Implement retry logic to reduce the likelihood of this issue affecting your operations.

Evaluate your application's reliability using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF

The impact times listed reflect the full duration of the incident and are not specific to individual customers. For advice on monitoring to understand detailed impact: https://aka.ms/AzPIR/Monitoring

Jim van Boven 0 Reputation points

2026-10-06T09:58:25.35+00:00

Dear reader,

From this morning, 6th of October, starting from about 04:59 UTC, about 83% of deployments from customers are failing. (50 of 60 AVD deployments, with varying amounts of VMs, (through the Azure Go SDK))

Was this issue resolved this summer? The deployments we checked are failing with internal server errors on the private endpoint connection deployments.

As of now, the issues have not been resolved. This afternoon, between 10-15 customers have student exams planned using AVD deployments. (up to 250 students in 1 exam)

Hope you can provide us with more information.
Andreas 0 Reputation points

2026-10-06T13:20:44.5833333+00:00

We had internal server errors all morning up until 15:00 CET