Windows Server Experiences Inconsistent Application Performance Due to NUMA Memory Allocation

Question

Windows Server Experiences Inconsistent Application Performance Due to NUMA Memory Allocation

Aisyah 25

Hi everyone

I'm experiencing an issue on a Windows Server system with multiple physical processors and a Non-Uniform Memory Access (NUMA) architecture, where an application shows inconsistent performance despite having sufficient available CPU and memory resources. During certain workloads, memory appears to be allocated from a remote NUMA node rather than the node where the application's threads are executing, resulting in increased memory access latency, reduced throughput, and unpredictable performance. The issue becomes more noticeable under heavy workloads or when multiple processes compete for system resources. However, overall CPU and memory utilization remain within normal ranges, making it difficult to identify the underlying bottleneck.

I'm trying to determine whether this behavior is related to Windows memory management, NUMA node affinity, thread scheduling, processor group configuration, or the application's memory allocation strategy.

How can I systematically investigate NUMA memory allocation behavior on Windows Server using Performance Monitor, Windows Performance Recorder (WPR), Windows Performance Analyzer (WPA), and relevant NUMA performance counters?

What diagnostic methods can help identify excessive remote memory access, incorrect processor affinity, or memory locality issues, and how can I optimize NUMA-aware scheduling and memory allocation without negatively affecting other applications or overall system stability?

Many thanks

0 comments

1 answer

Your answer

Answer 1

Hello,

This behavior is commonly associated with NUMA memory locality, where application threads execute on one NUMA node but access memory allocated on another. Remote memory access increases latency and can reduce throughput even when overall CPU and memory utilization appear normal.

The key is to investigate CPU placement, memory allocation, and thread scheduling together.

Monitor NUMA performance using Performance Monitor

Open perfmon.exe and monitor the following counters during normal and peak workloads:

\Processor Information(*)\% Processor Utility
\Memory\Available MBytes
\Memory\Pages/sec
\System\Processor Queue Length
\Process(*)\Working Set - Private

Also check the NUMA Node Memory performance object, where available, to compare memory availability across nodes.

You can identify available NUMA-related counters using:

Get-Counter -ListSet "*NUMA*"

Look for uneven memory pressure or CPU utilization across nodes. Standard Windows counters do not directly measure all remote DRAM accesses, so hardware performance counters may be necessary for confirmation.

Capture scheduling behavior using WPR and WPA

Use Windows Performance Recorder (WPR) to capture an ETW trace while the application experiences performance degradation. Open the resulting .etl file in Windows Performance Analyzer (WPA).

Examine CPU Usage (Sampled), CPU Usage (Precise), thread scheduling, and context switches. Look for frequent thread migration between NUMA nodes and scheduling delays.

Correlate these findings with hardware performance counters to confirm whether remote memory access is actually occurring.

Verify NUMA memory allocation and CPU affinity

For application-level investigation, Windows provides the following APIs:

GetNumaHighestNodeNumber();
GetNumaNodeProcessorMaskEx();
GetNumaAvailableMemoryNodeEx();
GetCurrentProcessorNumberEx();
GetThreadGroupAffinity();

For applications that require NUMA-aware memory allocation, VirtualAllocExNuma() allows specifying a preferred NUMA node. However, this preference does not guarantee that physical memory will always be allocated from that node.

Apply optimization carefully

Avoid immediately forcing CPU affinity or restricting memory allocation to a single NUMA node. These changes can increase contention and negatively affect other workloads.

Instead, allow Windows Scheduler to manage processor placement unless measurements demonstrate a specific problem. For NUMA-sensitive applications, coordinate thread placement and memory allocation, then compare throughput, latency, and remote memory traffic under representative workloads.

If the server is virtualized, also verify that the virtual NUMA topology is correctly aligned with the physical host.

The most reliable approach is to correlate per-node memory statistics, CPU scheduling traces, and hardware performance counters before changing affinity or memory allocation policies.

I hope you've found something useful here. If it helps you get more insight into the issue, it's appreciated to accept the answer. Should you have more questions, feel free to leave a message. Have a nice day!

Domic Vo.