AKS 1.36.4: cloud-node-manager and CoreDNS fail with "TLS handshake timeout" to API server after auto-upgrade; new nodes never initialize

After an AKS auto-upgrade (stable channel) moved our control plane and system node pool to 1.36.4 (centralus, Free tier, kubenet), the new versions of several AKS-managed kube-system add-ons can't connect to the API server. Every attempt fails with:

"Failed to wait for apiserver being healthy" err="... failed to get apiserver /healthz status: Get \"https://<cluster>.hcp.centralus.azmk8s.io:443/healthz\": net/http: TLS handshake timeout"

• Affected: cloud-node-manager v1.36.6 (CrashLoopBackOff on every node), CoreDNS v1.14.3-21 (new pods never ready), coredns-autoscaler, and the new metrics-server and KEDA metrics-apiserver pods. The pre-upgrade pods of the same add-ons still work.
• Impact: new nodes keep the node.cloudprovider.kubernetes.io/uninitialized taint and never get a providerID. The cluster autoscaler deletes them after 15 minutes, so our system pool has churned every ~45 minutes for 3 days. The user pool upgrade fails with PodDrainFailure, because metrics-server-pdb only has the one old healthy replica.
• Ruled out: from the same nodes, curl, openssl 3.5 (with and without X25519MLKEM768) and kubectl v1.36 (Go TLS) all complete the handshake to the same FQDN. API server and etcd metrics are flat (~13% CPU). There's no SNAT exhaustion, no proxy, no authorized IP ranges, and the cluster isn't private.

Has anyone seen managed add-ons fail TLS to the control plane after moving to 1.36, or does anyone know a safe way to get AKS to re-roll these add-ons? We don't have a paid support plan. Full details are here:

Describe the bug

After the AKS auto-upgrade (upgradeChannel: stable) moved our control plane and system pool to 1.36.4, the new versions of several AKS-managed kube-system add-ons can't connect to the API server. Every attempt fails with net/http: TLS handshake timeout. The previous versions of the same add-ons, still running from before the upgrade, connect fine.

Because cloud-node-manager can't reach the API server, no new node has initialized for 3 days. New nodes register and report Ready, but they keep the node.cloudprovider.kubernetes.io/uninitialized:NoSchedule taint and never get a spec.providerID. As a result:

• The cluster autoscaler counts each new system node as unregistered and deletes it after maxNodeProvisionTime (15m), then scales up again. Our system pool has looped like this about every 45 minutes since 2026-10-12 ~15:00Z: about 95 nodes created and deleted (instance names vmss00000k through vmss00003a). Each deletion fires a NodeNotReady alert, and the pool spends roughly 30 of every 45 minutes at zero nodes.
• The user pool upgrade to 1.36.4 fails with PodDrainFailure. Drain is blocked by metrics-server-pdb (minAvailable 1) because the two new metrics-server replicas crashloop with the same TLS timeout, leaving the single old replica as the only healthy one. The portal diagnostic reports a "restrictive PDB" as the root cause. We think that's a symptom: relaxing the PDB would evict the last working metrics-server, and the surge or replacement node wouldn't initialize anyway.

Affected components (all with the same TLS handshake timeout to https://<cluster>.hcp.centralus.azmk8s.io:443):

ComponentImage / versionStatecloud-node-managermcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.6-1CrashLoopBackOff on all nodes (~490 restarts / 45h)coredns (new ReplicaSet)mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.14.3-212 of 2 new pods 0/1 not ready for 45h (plugin/ready: Plugins not ready: "kubernetes")coredns-autoscaler(managed)CrashLoopBackOff (~1,380 restarts)metrics-server (new ReplicaSet)(managed)CrashLoopBackOff (~800 restarts)keda-operator-metrics-apiserver (new ReplicaSet)(managed)CrashLoopBackOff (~680 restarts)konnectivity-agent-autoscaler(managed)CrashLoopBackOffStill working (pre-upgrade ReplicaSets): coredns:v1.13.1-20 (1 pod, now serving all cluster DNS), the old metrics-server (1 pod) and the old keda-operator-metrics-apiserver pods.

cloud-node-manager log (identical on every restart and on every node):

I1015 14:08:32.211804  1 nodemanager.go:102] "Version" logger="Run" version="v1.36.6"
I1015 14:08:32.212933  1 azure.go:456] "Setting up ARM client factory for network resources" logger="InitializeCloudFromConfig" subscriptionID=""
I1015 14:08:32.213440  1 azure.go:464] "Setting up ARM client factory for compute resources" logger="InitializeCloudFromConfig" subscriptionID=""
I1015 14:08:32.214370  1 nodemanager.go:149] "Started cloud-node-manager" logger="startControllers"
E1015 14:09:03.272958  1 nodemanager.go:154] "Failed to wait for apiserver being healthy" err="timed out waiting for the condition: failed to get apiserver /healthz status: Get \"https://<cluster>.hcp.centralus.azmk8s.io:443/healthz\": net/http: TLS handshake timeout" logger="startControllers"

(We also noticed subscriptionID="" in the ARM client setup. We don't know whether that's expected.)

coredns-autoscaler:

W1015 13:31:16.847709  1 reflector.go:561] failed to list *v1.Node: Get "https://<cluster>.hcp.centralus.azmk8s.io:443/api/v1/nodes?limit=500&resourceVersion=0": net/http: TLS handshake timeout

Cluster autoscaler status for the system pool: registered.total: 0, unregistered: 1, scaleUp: InProgress.

What we ruled out

From the same nodes, at the same time the add-ons were failing:

• curl (pod network) to the FQDN and to kubernetes.default.svc: 16/16 succeed, TLS in about 20–100 ms (HTTP 401 as expected without credentials).
• openssl s_client 3.5 (hostNetwork), with both -groups X25519 and -groups X25519MLKEM768: all connect immediately. So it's not post-quantum ClientHello size or MTU.
• kubectl v1.36.0 (Go TLS stack; registry.k8s.io/kubectl:v1.36.0), from both hostNetwork and pod network, with and without GODEBUG=tlsmlkem=0: the handshake completes (401 Unauthorized with a dummy token).
• The control plane isn't overloaded. AKS platform metrics for 2026-10-10 through 10-15 are flat across the 10-12 start: apiserver_cpu_usage_percentage averages 13% (max 26%), apiserver_memory_usage_percentage is 6–7%, apiserver_current_inflight_requests averages 5 (max 15), and etcd CPU, memory and DB are all at 3% or less.
• No SNAT exhaustion. The outbound load balancer shows 0 failed SNAT connections from 10-10 through 10-15, and 3 of 6,144 allocated SNAT ports in use.
• kubectl from outside the cluster works normally. /readyz?verbose passes.
• No HTTP proxy configured, no authorized IP ranges, not a private cluster. Kubenet, outboundType: loadBalancer with one managed outbound IP. The node subnet route table is the AKS-managed kubenet route table. The only non-AKS webhook is the Microsoft billing-webhook (extensions usage, failurePolicy: Ignore).
• The kubelet on the new nodes registers fine. Only the add-on pods fail.

So the API server is reachable and handshakes complete for every client we try from the same node and network. Only the managed add-on pods time out, deterministically, on every attempt.

Timeline (UTC)

• Up to 2026-10-12 ~15:00: stable. A single system node had run for 7 days.
• 2026-10-12 ~15:00: the system node was replaced, and the create/delete loop began (initially with 1.35.8 nodes).
• 2026-10-13 ~16:30–17:31: control plane and system pool on 1.36.4. The current add-on pods date from here. The user pool upgrade failed with PodDrainFailure (drain attempts 16:59Z and 17:41Z).
• 2026-10-14 16:37: auto-upgrade retried, and the drain was blocked again on metrics-server-pdb.
• 2026-10-15: still ongoing. Sev1 NodeNotReady alerts every ~45 min.

Expected behavior

Managed add-ons connect to the API server after the upgrade, and new nodes initialize (providerID set, uninitialized taint removed).

Environment

• Region: centralus
• Kubernetes: control plane 1.36.4. systemnode8 1.36.4 (Standard_D4ads_v5, autoscaler min 1 / max 3). User pool aznode8cpu2 still on 1.35.8 (pool target 1.36.4, provisioning Failed, 2× Standard_D8as_v5).
• Node image: AKSAzureLinux-V3gen2-202610.03.0 (Azure Linux 3, kernel 6.6.157.1-1.azl3, containerd 2.2.4)
• Tier: Free (sku.tier: Free), supportPlan: KubernetesOfficial
• Auto-upgrade: upgradeChannel: stable, node OS channel not set
• Network: kubenet, Standard LB outbound
• Add-ons: Container Insights (ama-logs), Azure Monitor metrics, Key Vault Secrets Provider, KEDA, Azure Disk/File CSI

Additional context / ask

• We don't have a paid Azure support plan we can use for this, so we're hoping the AKS team can look at the control-plane side, or tell us what the managed add-ons are doing differently from a stock Go client.
• Happy to share the cluster resource ID, correlation IDs and full logs privately. Please let us know how to get them to you.
• Our current risk: cluster DNS depends on one remaining pre-upgrade CoreDNS pod. If it restarts, its replacement can't become ready. We are deliberately not retrying the upgrade, reconciling, or relaxing PDBs until we understand the cause.