[RCA 및 재발방지 요청] Azure VM 간 순간 통신 단절에 따른 Galera Cluster 반복 단절

Hello Jaeyeon Kim,

Thank you for providing the detailed history. We understand the concern, particularly because the communication interruptions have occurred repeatedly across both production and development Galera clusters.

The Galera messages showing TCP port 4567 communication time-outs confirm that cluster communication was interrupted. However, these application logs alone do not establish where the interruption originated. Possible layers include the guest operating system, VM resource pressure, NSG or firewall processing, user-defined routing, a network virtual appliance, packet loss or latency on the network path, or an Azure host event. A timestamp-specific review is required before attributing the incidents to Azure maintenance, live migration, or another platform operation.

Azure platform maintenance normally uses the least disruptive mechanism available. Depending on the operation, a VM can be briefly paused or live-migrated, with network connections intended to be preserved. Even a short interruption may nevertheless be detected by a latency-sensitive clustered application if its membership or heartbeat thresholds are exceeded. Review Azure VM maintenance behaviour [Maintenanc...soft Learn | Learn.Microsoft.com]

Because the request covers seven historical incidents from March 2025 through October 2026, a public forum cannot access the subscription-level host and network telemetry required to provide a definitive consolidated RCA. Please create or continue an Azure support request and provide the following for each occurrence:

• Exact start and recovery timestamps in UTC.
• Subscription ID, resource group, region, VM names and VM resource IDs.
• Galera logs from every node covering at least 15 minutes before and after the event.
• Operating-system logs and performance data for CPU, memory, disk and network during the same interval.
• Network topology, including VNets, subnets, NSGs, UDRs, Azure Firewall or third-party virtual appliances, load balancers and VPN/ExpressRoute paths.
• Whether Accelerated Networking is enabled on every cluster VM.
• VM Resource** Health, Activity Log, and Service **Health records for the reported timestamps.
• Bidirectional packet captures, where available.

For recurrence detection, configure Azure Network Watcher Connection Monitor between every Galera node on TCP 4567. Connection Monitor can record connectivity, latency, packet loss and status over time. Network Watcher also supports packet capture and network-flow visibility, which can provide evidence if another intermittent event occurs. [Network Wa...soft Azure | Azure.Microsoft.com], [Visualize...Azure Blog | Azure.Microsoft.com]

We also recommend:

1. Place production nodes across Availability Zones where the region and selected VM size support them, while accounting for inter-zone latency.
2. Keep cluster traffic on private IP paths and minimise unnecessary inspection, NAT or routing appliances between nodes.
3. Confirm NSGs, guest firewalls and intermediate devices allow bidirectional Galera communication, including TCP/UDP 4567 and any other ports required by the configured Galera functions.
4. Verify that VM sizes provide sufficient network bandwidth and that Accelerated Networking is enabled where supported.
5. Review Galera membership, heartbeat and eviction parameters with the database vendor before changing them. Thresholds should tolerate brief network variation without delaying detection of genuine node failures.
6. Add application-level monitoring and recovery procedures so the database service remains available if a node temporarily leaves the primary component.
7. Do not rely on a single VM or single datacentre failure boundary for a production cluster.

Availability Zones provide separate datacentre failure domains, while Azure Service Health, Azure Monitor and Network Watcher provide platform, resource and network visibility. [Azure Reli...soft Azure | Azure.Microsoft.com]

At this stage, the evidence supports a cluster** communication **timeout, but it does not conclusively identify an Azure host, maintenance or network-platform event as the root cause. Microsoft Support can correlate each exact timestamp with the retained platform telemetry and determine whether the incidents share a common platform correlation. If the relevant historical telemetry is no longer available, the result may be limited to a best-effort assessment; therefore, proactive Connection Monitor and packet-capture configuration is important for any future occurrence.Hello Jaeyeon Kim,

Thank you for providing the detailed history. We understand the concern, particularly because the communication interruptions have occurred repeatedly across both production and development Galera clusters.

The Galera messages showing TCP port 4567 communication time-outs confirm that cluster communication was interrupted. However, these application logs alone do not establish where the interruption originated. Possible layers include the guest operating system, VM resource pressure, NSG or firewall processing, user-defined routing, a network virtual appliance, packet loss or latency on the network path, or an Azure host event. A timestamp-specific review is required before attributing the incidents to Azure maintenance, live migration, or another platform operation.

Azure platform maintenance normally uses the least disruptive mechanism available. Depending on the operation, a VM can be briefly paused or live-migrated, with network connections intended to be preserved. Even a short interruption may nevertheless be detected by a latency-sensitive clustered application if its membership or heartbeat thresholds are exceeded. Review Azure VM maintenance behaviour [Maintenanc...soft Learn | Learn.Microsoft.com]

Because the request covers seven historical incidents from March 2025 through October 2026, a public forum cannot access the subscription-level host and network telemetry required to provide a definitive consolidated RCA. Please create or continue an Azure support request and provide the following for each occurrence:

• Exact start and recovery timestamps in UTC.
• Subscription ID, resource group, region, VM names and VM resource IDs.
• Galera logs from every node covering at least 15 minutes before and after the event.
• Operating-system logs and performance data for CPU, memory, disk and network during the same interval.
• Network topology, including VNets, subnets, NSGs, UDRs, Azure Firewall or third-party virtual appliances, load balancers and VPN/ExpressRoute paths.
• Whether Accelerated Networking is enabled on every cluster VM.
• VM Resource** Health, Activity Log, and Service **Health records for the reported timestamps.
• Bidirectional packet captures, where available.

For recurrence detection, configure Azure Network Watcher Connection Monitor between every Galera node on TCP 4567. Connection Monitor can record connectivity, latency, packet loss and status over time. Network Watcher also supports packet capture and network-flow visibility, which can provide evidence if another intermittent event occurs. [Network Wa...soft Azure | Azure.Microsoft.com], [Visualize...Azure Blog | Azure.Microsoft.com]

We also recommend:

1. Place production nodes across Availability Zones where the region and selected VM size support them, while accounting for inter-zone latency.
2. Keep cluster traffic on private IP paths and minimise unnecessary inspection, NAT or routing appliances between nodes.
3. Confirm NSGs, guest firewalls and intermediate devices allow bidirectional Galera communication, including TCP/UDP 4567 and any other ports required by the configured Galera functions.
4. Verify that VM sizes provide sufficient network bandwidth and that Accelerated Networking is enabled where supported.
5. Review Galera membership, heartbeat and eviction parameters with the database vendor before changing them. Thresholds should tolerate brief network variation without delaying detection of genuine node failures.
6. Add application-level monitoring and recovery procedures so the database service remains available if a node temporarily leaves the primary component.
7. Do not rely on a single VM or single datacentre failure boundary for a production cluster.

Availability Zones provide separate datacentre failure domains, while Azure Service Health, Azure Monitor and Network Watcher provide platform, resource and network visibility. [Azure Reli...soft Azure | Azure.Microsoft.com]

At this stage, the evidence supports a cluster** communication **timeout, but it does not conclusively identify an Azure host, maintenance or network-platform event as the root cause. Microsoft Support can correlate each exact timestamp with the retained platform telemetry and determine whether the incidents share a common platform correlation. If the relevant historical telemetry is no longer available, the result may be limited to a best-effort assessment; therefore, proactive Connection Monitor and packet-capture configuration is important for any future occurrence.