Production AKS System Nodes NotReady – OverconstrainedZonalAllocationRequest – Engineering Assistance Required

Question

Production AKS System Nodes NotReady – OverconstrainedZonalAllocationRequest – Engineering Assistance Required

Ankit Bisht 0

Production AKS System Nodes NotReady – OverconstrainedZonalAllocationRequest – Engineering Assistance Required

Hello Microsoft AKS Support Team,

We are experiencing an ongoing issue with our production Azure Kubernetes Service (AKS) cluster in West Europe, which has remained unresolved since 24 October 2026.

Both system node pool instances are NotReady, and the underlying Azure Virtual Machine Scale Set (VMSS) is reporting allocation failures.

Although our user node pool remains operational, the cluster is currently running without healthy system-node capacity.

1. Environment Details

Component	Configuration
Azure Region	West Europe
--------	--------
Azure Region	West Europe
Kubernetes Version	1.28.9
System Node Pool	`agentpool`
System Node VM Size	`Standard_DC2s_v2`
System Node Count	2
System Availability Zones	1, 2, 3
User Node Pool	`prdgeneral`
User Node VM Size	`Standard_DS3_v2`
User Node Count	3
User Availability Zone	3

2. Current Issue

Since 24 October 2026, both system nodes have remained in the NotReady state.

Kubernetes reports:

Node condition: NodeStatusUnknown

Reason: Kubelet stopped posting node status

Both system nodes are unavailable.

The underlying VMSS reports:

Error: OverconstrainedZonalAllocationRequest

Provisioning State: Failed

Latest Model Applied: False

The Azure allocation error indicates that the VM instances cannot be allocated with the existing combination of VM size, availability zone, and networking constraints.

3. AKS Node Pool and VMSS Configuration Mismatch

During troubleshooting, we attempted to change the underlying VMSS model to Standard_D2s_v5.

The current configuration is:

Component	VM Size
AKS-managed system node pool	`Standard_DC2s_v2`
--------	--------
AKS-managed system node pool	`Standard_DC2s_v2`
Underlying VMSS model	`Standard_D2s_v5`
Existing VMSS instances	`Standard_DC2s_v2`

Both existing instances report LatestModelApplied=false.

The direct VMSS model modification was performed after the original incident began and was not the initial cause of the outage.

We have stopped further direct VMSS modifications and require guidance on safely reconciling this mismatch through a Microsoft-supported procedure.

4. Production Impact

Our three user nodes remain operational and continue hosting production applications and critical Kubernetes components.

Current utilization:

Resource	Utilization
CPU	Approximately 7–8% per node
--------	--------
CPU	Approximately 7–8% per node
Memory	Approximately 60–76% per node

Although production applications are currently operational, the absence of healthy system nodes creates an ongoing availability and recovery risk.

Due to budget constraints, increasing the number of nodes is not feasible.

Our priority is to recover the existing infrastructure without disrupting healthy production workloads or introducing additional infrastructure costs.

5. Troubleshooting Already Completed

We have performed the following investigations:

Verified Kubernetes node conditions and identified both system nodes as NotReady.

Inspected the underlying VMSS and confirmed failed provisioning states.

Identified OverconstrainedZonalAllocationRequest as the reported Azure allocation failure.

Reviewed the AKS node pool and VMSS configurations.

Attempted system node pool scaling, which failed.

Attempted replacement node pool creation, which was blocked due to the unsupported Kubernetes version.

Checked available AKS upgrade targets.

The cluster currently runs Kubernetes 1.28.9, while Azure offers 1.34.x as an upgrade target.

We have not attempted the Kubernetes upgrade because we require confirmation that the procedure is supported and will not disrupt our healthy production nodes.

6. Assistance Required from Microsoft

We request assistance from the Microsoft AKS Engineering and Azure Compute/VMSS teams with the following questions:

1. System Node Pool Recovery

What is the recommended Microsoft-supported procedure to recover the existing system node pool when both VMSS instances are unavailable due to OverconstrainedZonalAllocationRequest?

2. VMSS Configuration Reconciliation

How can we safely reconcile the AKS-managed node pool configuration (Standard_DC2s_v2) with the underlying VMSS model (Standard_D2s_v5) without disrupting the existing production workloads?

3. Kubernetes Upgrade Path

Is a control-plane-only upgrade from Kubernetes 1.28.9 to 1.34.x supported in this recovery scenario?

If so, would it allow us to create a replacement system node pool while retaining the existing user nodes?

4. Alternative Recovery Procedure

If the control-plane upgrade is unsupported or unsafe, what alternative recovery approach does Microsoft recommend?

5. Azure Compute Allocation Investigation

Can Microsoft investigate whether the persistent allocation failures are caused by:

Regional or zonal VM SKU capacity limitations

Availability zone restrictions

VMSS networking constraints

Other Azure Compute allocation restrictions

7. Request for Engineering Escalation

We have already completed the initial troubleshooting recommendations, but the issue remains unresolved.

Our Azure subscription uses the Developer Support Plan, and our existing support request has not yet resulted in a validated recovery procedure.

Given the ongoing production impact, we request review by a Microsoft AKS or Azure Compute engineer and guidance on the available private technical escalation process.

We can provide the following information privately if required:

Azure subscription and resource identifiers

AKS and VMSS diagnostic outputs

Azure Activity Logs

Failed operation and correlation IDs

Additional cluster configuration details

Our primary requirement is a validated, step-by-step recovery procedure that restores system-node capacity without disrupting our healthy production workloads.

Thank you for your assistance.

Production AKS System Nodes NotReady – OverconstrainedZonalAllocationRequest – Engineering Assistance Required

Hello Microsoft AKS Support Team,

We are experiencing an ongoing issue with our production Azure Kubernetes Service (AKS) cluster in West Europe, which has remained unresolved since 24 October 2026.

Both system node pool instances are NotReady, and the underlying Azure Virtual Machine Scale Set (VMSS) is reporting allocation failures.

Although our user node pool remains operational, the cluster is currently running without healthy system-node capacity.

1. Environment Details

Component	Configuration
Azure Region	West Europe
Kubernetes Version	1.28.9
System Node Pool	`agentpool`
System Node VM Size	`Standard_DC2s_v2`
System Node Count	2
System Availability Zones	1, 2, 3
User Node Pool	`prdgeneral`
User Node VM Size	`Standard_DS3_v2`
User Node Count	3
User Availability Zone	3

2. Current Issue

Since 24 October 2026, both system nodes have remained in the NotReady state.

Kubernetes reports:

Node condition: NodeStatusUnknown

Reason: Kubelet stopped posting node status

Both system nodes are unavailable.

The underlying VMSS reports:

Error: OverconstrainedZonalAllocationRequest

Provisioning State: Failed

Latest Model Applied: False

The Azure allocation error indicates that the VM instances cannot be allocated with the existing combination of VM size, availability zone, and networking constraints.

3. AKS Node Pool and VMSS Configuration Mismatch

During troubleshooting, we attempted to change the underlying VMSS model to Standard_D2s_v5.

The current configuration is:

Component	VM Size
AKS-managed system node pool	`Standard_DC2s_v2`
Underlying VMSS model	`Standard_D2s_v5`
Existing VMSS instances	`Standard_DC2s_v2`

Both existing instances report LatestModelApplied=false.

The direct VMSS model modification was performed after the original incident began and was not the initial cause of the outage.

We have stopped further direct VMSS modifications and require guidance on safely reconciling this mismatch through a Microsoft-supported procedure.

4. Production Impact

Our three user nodes remain operational and continue hosting production applications and critical Kubernetes components.

Current utilization:

Resource	Utilization
CPU	Approximately 7–8% per node
Memory	Approximately 60–76% per node

Although production applications are currently operational, the absence of healthy system nodes creates an ongoing availability and recovery risk.

Due to budget constraints, increasing the number of nodes is not feasible.

Our priority is to recover the existing infrastructure without disrupting healthy production workloads or introducing additional infrastructure costs.

5. Troubleshooting Already Completed

We have performed the following investigations:

Verified Kubernetes node conditions and identified both system nodes as NotReady.

Inspected the underlying VMSS and confirmed failed provisioning states.

Identified OverconstrainedZonalAllocationRequest as the reported Azure allocation failure.

Reviewed the AKS node pool and VMSS configurations.

Attempted system node pool scaling, which failed.

Attempted replacement node pool creation, which was blocked due to the unsupported Kubernetes version.

Checked available AKS upgrade targets.

The cluster currently runs Kubernetes 1.28.9, while Azure offers 1.34.x as an upgrade target.

We have not attempted the Kubernetes upgrade because we require confirmation that the procedure is supported and will not disrupt our healthy production nodes.

6. Assistance Required from Microsoft

We request assistance from the Microsoft AKS Engineering and Azure Compute/VMSS teams with the following questions:

1. System Node Pool Recovery

What is the recommended Microsoft-supported procedure to recover the existing system node pool when both VMSS instances are unavailable due to OverconstrainedZonalAllocationRequest?

2. VMSS Configuration Reconciliation

How can we safely reconcile the AKS-managed node pool configuration (Standard_DC2s_v2) with the underlying VMSS model (Standard_D2s_v5) without disrupting the existing production workloads?

3. Kubernetes Upgrade Path

Is a control-plane-only upgrade from Kubernetes 1.28.9 to 1.34.x supported in this recovery scenario?

If so, would it allow us to create a replacement system node pool while retaining the existing user nodes?

4. Alternative Recovery Procedure

If the control-plane upgrade is unsupported or unsafe, what alternative recovery approach does Microsoft recommend?

5. Azure Compute Allocation Investigation

Can Microsoft investigate whether the persistent allocation failures are caused by:

Regional or zonal VM SKU capacity limitations

Availability zone restrictions

VMSS networking constraints

Other Azure Compute allocation restrictions

7. Request for Engineering Escalation

We have already completed the initial troubleshooting recommendations, but the issue remains unresolved.

Our Azure subscription uses the Developer Support Plan, and our existing support request has not yet resulted in a validated recovery procedure.

Given the ongoing production impact, we request review by a Microsoft AKS or Azure Compute engineer and guidance on the available private technical escalation process.

We can provide the following information privately if required:

Azure subscription and resource identifiers

AKS and VMSS diagnostic outputs

Azure Activity Logs

Failed operation and correlation IDs

Additional cluster configuration details

Our primary requirement is a validated, step-by-step recovery procedure that restores system-node capacity without disrupting our healthy production workloads.

Thank you for your assistance.

Ankit Bisht 0 Reputation points

2026-10-30T10:38:14.3733333+00:00

Thank you for the recommendations.

We have already investigated the VM allocation failure and reviewed the AKS node pool and underlying VMSS configuration.

The issue remains unresolved:

Both system nodes are NotReady.

The VMSS reports OverconstrainedZonalAllocationRequest.

The VMSS provisioning state is Failed.

The AKS node pool specifies Standard_DC2s_v2, while the VMSS model currently specifies Standard_D2s_v5. Existing instances remain on Standard_DC2s_v2.

Kubernetes 1.28.9 prevents us from creating a replacement node pool through the previously attempted procedure.

Our three healthy user nodes are currently supporting production workloads, and increasing infrastructure capacity is not feasible due to budget constraints.

Since this involves a persistent Azure Compute allocation failure and an AKS-managed VMSS configuration mismatch, could a Microsoft AKS or Azure Compute support engineer please review this case?

We specifically need a Microsoft-supported recovery procedure that does not disrupt our existing production workloads.

We can provide operation IDs, correlation IDs, and additional diagnostic information privately.

The issue has been ongoing since 24 October 2026, and our production cluster continues to operate without healthy system-node capacity.

We would appreciate assistance from a Microsoft engineer rather than additional generic troubleshooting recommendations.

1 answer

Your answer

Ankit Bisht 0 Reputation points

2026-10-30T10:38:14.3733333+00:00

Thank you for the recommendations.

We have already investigated the VM allocation failure and reviewed the AKS node pool and underlying VMSS configuration.

The issue remains unresolved:

Both system nodes are NotReady.

The VMSS reports OverconstrainedZonalAllocationRequest.

The VMSS provisioning state is Failed.

The AKS node pool specifies Standard_DC2s_v2, while the VMSS model currently specifies Standard_D2s_v5. Existing instances remain on Standard_DC2s_v2.

Kubernetes 1.28.9 prevents us from creating a replacement node pool through the previously attempted procedure.

Our three healthy user nodes are currently supporting production workloads, and increasing infrastructure capacity is not feasible due to budget constraints.

Since this involves a persistent Azure Compute allocation failure and an AKS-managed VMSS configuration mismatch, could a Microsoft AKS or Azure Compute support engineer please review this case?

We specifically need a Microsoft-supported recovery procedure that does not disrupt our existing production workloads.

We can provide operation IDs, correlation IDs, and additional diagnostic information privately.

The issue has been ongoing since 24 October 2026, and our production cluster continues to operate without healthy system-node capacity.

We would appreciate assistance from a Microsoft engineer rather than additional generic troubleshooting recommendations.

Answer 1

For OverconstrainedZonalAllocationRequest, the supported guidance in the provided material is to treat this as an Azure Compute allocation constraint issue, not as a Kubernetes control-plane issue.

Recommended path:

Identify the exact failed AKS or node pool operation and capture the error details.

Check cluster state:

   az aks show \
     --resource-group <resource-group-name> \
     --name <cluster-name> \
     --query "{ProvisioningState:provisioningState, PowerState:powerState.code, NodeProvisioningMode:nodeProvisioningProfile.mode, AutoUpgradeChannel:autoUpgradeProfile.upgradeChannel, NodeOSUpgradeChannel:autoUpgradeProfile.nodeOSUpgradeChannel}" \
     --output table

Check the affected system node pool state:

   az aks nodepool show \
     --resource-group <resource-group-name> \
     --cluster-name <cluster-name> \
     --name agentpool \
     --query "{ProvisioningState:provisioningState, PowerState:powerState.code, PoolType:typePropertiesType, Count:count, AutoscalerEnabled:enableAutoScaling, MinCount:minCount, MaxCount:maxCount}" \
     --output table

Retrieve the latest failed operation for the cluster or node pool:

   az aks operation show-latest \
     --resource-group <resource-group-name> \
     --name <cluster-name> \
     --nodepool-name agentpool \
     --query "{OperationId:name, Status:status, StartTime:startTime, EndTime:endTime, Error:error}" \
     --output json

This confirms whether the failure is on the AKS agent pool resource and captures the operation error and timestamps.

Query the Azure activity log for the failed resource.

Get the node pool resource ID:

   az aks nodepool show \
     --resource-group <resource-group-name> \
     --cluster-name <cluster-name> \
     --name agentpool \
     --query id --output tsv

Use that ID in the failed activity log query:

   az monitor activity-log list \
     --resource-id <target-resource-id> \
     --offset 24h \
     --status Failed \
     --query "[].{Time:eventTimestamp, Operation:operationName.localizedValue, Caller:caller, Error:properties.statusMessage, EventId:eventDataId, ActivityLogOperationId:operationId, CorrelationId:correlationId}" \
     --output json

Record the caller, error code, subcode, message, operation ID, event ID, and correlation ID. Those are the key artifacts for escalation.

For OverconstrainedAllocationRequest or OverconstrainedZonalAllocationRequest, use the documented recovery options.
- If a compatible VM size has capacity, resize the existing VMSS/VM node pool in place to another size. This is documented as a preview feature and requires surge capacity for the target VM size.
- If the workload is zone-spanning, use automatic zone placement so AKS can select zones with capacity for the requested SKU.
- If the error is caused by too many constraints, one documented mitigation is to create a new node pool that is not associated with a proximity placement group.
Do not rely on a control-plane-only upgrade as the recovery method for this allocation error. The provided material does not state that a control-plane-only upgrade from 1.28.9 to 1.34.x is a supported fix for OverconstrainedZonalAllocationRequest, and it does not provide support guidance for skipping directly between those versions. The supported guidance in the material is to resolve the failed resource or allocation constraint first.
For portal-based validation, use:
- AKS cluster > Diagnose and solve problems > Provisioning State Check
- AKS cluster > Node pools to inspect the affected pool state
- AKS cluster > Activity log filtered to Failed

About the VMSS configuration mismatch:

The provided material supports checking the AKS node pool state, the latest AKS operation, and the failed activity log.
It does not provide a supported procedure for manually reconciling an AKS-managed VMSS model mismatch between the node pool definition and existing instances.
Because the failure is persistent, affects the system pool, and the requested engineering review depends on operation IDs and correlation IDs, the supported next step from the available guidance is to collect those diagnostics and use them for escalation.