AKS: Azure-provided DNS (168.63.129.16) intermittently unreachable from a single node for ~1 minute (Israel Central)

Gur Lurye 0 Reputation points
2026-11-16T08:12:24.0066667+00:00

We run an AKS cluster (Kubernetes 1.35, Ubuntu 24.04 nodes) in Israel Central,

using the default Azure-provided DNS (no custom VNet DNS servers).

On 2026-11-16 between 04:25:53 and 04:26:58 UTC, one of our two CoreDNS

replicas logged ~100 upstream timeouts like:

[ERROR] plugin/errors: 2 <name>. A: read udp <pod-ip>:<port>->168.63.129.16:53: i/o timeout

Every name it forwarded failed (Azure Redis, Azure Postgres, Storage,

graph.microsoft.com, public domains). The other CoreDNS replica, on a

different node, had zero errors in the same window. Resolution recovered

by itself after ~65 seconds. We saw identical symptoms on 2026-10-21

around 12:41 UTC.

What we ruled out:

  • Throttling: ~12 queries/sec per CoreDNS replica
  • Node/pod restarts: nodes Ready continuously since September, 0 CoreDNS restarts
  • Upgrades: auto-upgrade and node OS upgrade channels are disabled
  • Config changes: none on our side in that window
  • No custom DNS, firewall or NVA in the DNS path

The only related signal: Resource Health reported a PlatformInitiated

"Downtime" (1 of 2 VMs unavailable) on another node pool's VMSS in the same

cluster at 04:09–04:16 UTC, about 10 minutes earlier. No Service Health

notices.

Questions:

  1. Has anyone else seen short 168.63.129.16 outages in Israel Central around these times?
  2. Is a host-level loss of Azure DNS for ~1 minute expected during platform maintenance, and is there any way to get notified of it?
  3. Is AKS LocalDNS with serve-stale the recommended mitigation, or is there something else we should do?
Azure Kubernetes Service
Azure Kubernetes Service

An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.

0 comments No comments

1 answer

Sort by: Most helpful
  1. AI answer

    2026-11-16T08:13:34.6233333+00:00

    No public Azure incident currently matches AKS DNS failures in Israel Central at either timestamp, so enable AKS LocalDNS with stale-response caching as the resilience measure and open an Azure support case to correlate the affected node with host, SDN, and platform-DNS telemetry.

    What the evidence indicates

    • [IP Address] is Azure’s fixed platform virtual IP and provides Azure-provided DNS. It is a virtual IP of the host node, is not subject to user-defined routes, and DNS traffic is not subject to NSGs by default unless explicitly targeted through the AzurePlatformDNS service tag.^1^
    • One CoreDNS replica timing out while the replica on another node remained healthy strongly narrows the failure to the affected node’s path to the platform DNS VIP. It does not, by itself, establish whether the cause was the host, Azure fabric, or the node OS networking stack.
    • The public Azure status history does not identify an incident matching AKS, Israel Central, [IP Address], and the two reported windows.^2^
    • The earlier Resource Health event on a different VMSS is relevant supporting evidence, but it does not prove that the later DNS interruption had the same cause. Resource Health is resource-specific, whereas Service Health reports known broader platform events.^3^

    A one-minute loss of platform DNS is not documented as normal maintenance behavior. Maintenance that does not require a reboot can pause a VM for a few seconds; reboot-requiring maintenance is notified through Service Health. Those documented behaviors do not account for this event conclusively.^4^

    Microsoft recommends LocalDNS to improve AKS DNS reliability. It runs a resolver on each node, reduces dependency on centralized CoreDNS paths, and can serve cached records while an upstream resolver is temporarily unavailable. Stale serving is best-effort: a response must already be present in that node’s cache, and cache eviction can still cause misses.^5^

    For AKS Standard on Kubernetes 1.35:

    1. Confirm that the affected node-pool VM SKU has at least 4 vCPUs. Kubernetes 1.35 and Ubuntu 24.04 satisfy the documented version and OS prerequisites; VM capacity remains the unstated prerequisite.
    2. Create a node-pool LocalDNS JSON profile using the documented default template.
    3. Keep the root-zone forwarding destination as VnetDNS.
    4. Configure:
      • serveStale: Immediate
      • serveStaleDurationInSeconds: an interval appropriate for the workload; the documented default is 3600
      • cacheDurationInSeconds: based on record volatility; it caps rather than extends the upstream TTL
      • queryLogging: initially Log if additional evidence is required
    5. Apply the profile to a nonproduction node pool first, then use Required mode for the production pool after validation. On Kubernetes 1.31–1.36, Preferred only validates the configuration and does not enable LocalDNS.
    6. Schedule the change carefully: changing the LocalDNS profile on an existing managed node pool triggers a node reimage. Ensure replicas and Pod Disruption Budgets tolerate the rollout.^6^

    Do not treat LocalDNS as proof or correction of the underlying platform-path issue; it limits the impact of short outages for records that are cached.

    Notifications

    Configure both alert types because they cover different scopes:

    1. Service Health alert
      • Azure portal → Service Health → Service issues → Create service health alert
      • Scope: affected subscription
      • Region: Israel Central
      • Event types: Service issues and Planned maintenance
      • Services: include Virtual Machine Scale Sets, Virtual Machines, and relevant AKS services
      • Attach a global Action Group for email, SMS, webhook, Logic App, or ITSM notification.^7^
    2. Resource Health alert
      • Target the AKS node-pool VM scale sets.
      • Alert on Unavailable, Degraded, and Unknown.
      • Include Platform initiated reason type and Active/Resolved transitions.^3^

    These alerts only fire when Azure publishes the corresponding health event; they do not provide a guaranteed notification for every brief DNS-only interruption.

    Evidence for the next occurrence

    Run an always-on or quickly deployable DNS trace filtered to the platform resolver:

    kubectl gadget run trace_dns:$GADGET_VERSION \
      --all-namespaces \
      --fields src,dst,id,qr,name,nameserver,rcode,latency_ns \
      --filter "nameserver.addr==[IP Address]"
    

    A query with no matching response demonstrates that the request was observed without a DNS reply.^8^

    For the support case, provide:

    • Exact UTC windows, including both previous occurrences
    • Cluster, node-pool VMSS, affected node, CoreDNS pod and pod IP
    • CoreDNS logs and query counts
    • Trace showing requests to [IP Address] without replies
    • Node kernel/network logs covering several minutes before and after
    • Resource Health event details from all node-pool VMSS resources
    • Confirmation that the peer CoreDNS node was unaffected

    Request correlation against host networking, SDN/fabric, VMSS maintenance, and platform DNS telemetry. That backend correlation is required to establish whether this was a host-level Azure event.


    References

    1. Azure IP Address [IP Address] Overview | Microsoft Learn
    2. Azure status history | Microsoft Azure
    3. Resource Health alerts
    4. Handling planned maintenance notifications
    5. Best practices for network connectivity and security in Azure Kubernetes Service (AKS)
    6. Configure LocalDNS in Azure Kubernetes Service (AKS)
    7. Create Service Health alerts
    8. Troubleshoot Azure Kubernetes Service DNS failures in real time
    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.