Kubernetes Service (AKS): Cluster or NodePool shows in Failed State — Issue provisioning NC4as_T4_v3 nodes in East US 2

Question

Kubernetes Service (AKS): Cluster or NodePool shows in Failed State — Issue provisioning NC4as_T4_v3 nodes in East US 2

Steph-0274 0

## Problem description I am experiencing an AKS node pool provisioning and replacement issue in East US 2. The affected GPU node pool was originally running successfully with the following configuration:

VM size: Standard_NC4as_T4_v3
OS: Linux/Ubuntu
OS disk: Ephemeral
Availability zones: not specified
Networking: Azure CNI
Node count: 1
Kubernetes version after the upgrade: 1.35.5
GPU workload label and taint configured for scheduling GPU workloads

During the Kubernetes upgrade, AKS attempted to replace the existing GPU node. The replacement failed, and the node pool entered a Failed state. This is particularly concerning because the original node had been running successfully before the upgrade. The issue appears to be related to the allocation of the replacement VM rather than to the workload itself.

## Environment

Azure Kubernetes Service
Region: East US 2
Resource type: Microsoft.ContainerService/managedClusters
Networking: Azure CNI
GPU VM family: NCas T4 v3
Original VM size: Standard_NC4as_T4_v3
Kubernetes version: 1.35.5
Node pool size: 1 node

## What I have already tried I tested several configurations in an attempt to determine whether the issue was related to the VM size, disk type, or availability zone:

Standard_NC4as_T4_v3 with Managed OS disk
Standard_NC8as_T4_v3 with Managed OS disk
Standard_NC8as_T4_v3 with Ephemeral OS disk
Deployments without specifying an availability zone
Deployments in availability zones 1, 2, and 3
Recreating the original GPU node pool with Managed OS disk

The attempts failed with errors including: "OverconstrainedAllocationRequest" "Allocation failed. VM(s) with the following constraints cannot be allocated because the condition is too restrictive." Constraints:

Networking Constraints
VM Size

When using Ephemeral disks, the error also included:

Differencing (Ephemeral) Disks
Networking Constraints
VM Size Some zonal attempts returned:

ZonalAllocationFailed There is not sufficient capacity for the requested VM size in this zone.

## Current status The cluster itself has been reporting Succeeded, but the GPU node pool entered a Failed state during the replacement and recreation attempts.

## Questions

Is this issue specific to GPU VM capacity in East US 2, or can it affect the NCas T4 v3 family in other regions and availability zones?
How can we verify GPU VM capacity and allocation availability before performing an AKS node pool upgrade?
Is this a known limitation involving AKS upgrades, GPU VM sizes, Azure CNI, Accelerated Networking, or Ephemeral OS disks?
What is the recommended and supported workaround to safely upgrade or replace a single GPU node without risking workload disruption?
Should we use a separate GPU node pool or Capacity Reservation before upgrading production?
What configuration and upgrade strategy does Microsoft recommend for our production GPU node pools?

0 comments

2 answers

Your answer

Answer 1

Hello Steph-0274

Based on the errors and troubleshooting you've already performed, this looks like an Azure Compute capacity/placement constraint affecting the NCas T4 v3 GPU family in East US 2.

Both NC4as_T4_v3 and NC8as_T4_v3 fail, including configurations without availability zones and with Managed OS disks.

OverconstrainedAllocationRequest: Azure cannot allocate VMs satisfying the combined SKU, networking, disk, and placement constraints.
ZonalAllocationFailed: Insufficient capacity for the requested VM size in the specified availability zone.

These errors are not exclusive to East US 2 or GPU VMs.

I recommend:

Check quota separately from capacity. Use az vm list-usage --location eastus2 and az vm list-skus --location eastus2 --size Standard_NC4as_T4_v3 --all. These help identify quota and SKU restrictions, but neither guarantees live VM allocation capacity.
Avoid further deletion or recreation of the existing GPU pool until replacement capacity is confirmed. Since Managed OS disks and different zone configurations also failed, changing disk settings alone is unlikely to resolve this.
Try creating a second GPU node pool using an alternative GPU SKU supported by AKS in East US 2. Validate that a node becomes Ready, NVIDIA drivers work, and the GPU workload runs successfully before migrating workloads.
For production, avoid relying on a single GPU node. Use a second GPU node pool or reserve sufficient GPU capacity where supported. Microsoft supports associating Capacity Reservation Groups with new AKS node pools.
Open an Azure Support case under AKS/VM allocation and request investigation by the Azure Compute capacity team. Provide the failed VMSS deployment IDs, timestamps, region, attempted SKUs, and exact allocation errors. Ask whether NCas T4 v3 capacity is currently constrained for your subscription in East US 2.

For upgrades where additional GPU capacity is unavailable, AKS also supports maxSurge=0 with maxUnavailable, allowing in-place upgrades. I would not recommend that approach for a single production GPU node, because it requires downtime and may still fail if Azure cannot allocate the replacement VM.

Confirm alternative GPU capacity first, deploy and validate a replacement pool, migrate workloads, and only then retire the failed pool. For production upgrades, consider capacity reservations to reduce the risk of encountering the same allocation failure again.

Documentation

Answer 2

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more