An Azure service that automates the access and use of data across clouds without writing code.
Standard unit testing framework fails on Linux: DllNotFoundException for kernel32.dll (GlobalMemoryStatusEx), then tests hang
Describe the bug
On Linux, Logic Apps Standard unit tests created with the VS Code extension's testing framework (Microsoft.Azure.Workflows.WebJobs.Tests.Extension) fail or hang. On Windows the same tests pass. Running the unit tests in a CI pipeline on a Linux agent is therefore not possible, and the documentation doesn't mention any operating system requirement.
The workflow engine reads memory utilisation through kernel32!GlobalMemoryStatusEx, which doesn't exist on Linux:
Microsoft.Azure.Workflows.Common.ErrorResponses.ErrorResponseMessageException: Workflow processing failed. Error: 'Workflow '<id>' is unhealthy. Error:'System.DllNotFoundException: Unable to load shared library 'kernel32.dll' or one of its dependencies. ...
<test bin>/runtimes/linux-x64/native/kernel32.dll.so: cannot open shared object file: No such file or directory
...
at Microsoft.Azure.Workflows.Common.PerformanceCounters.WebAppWorkerMemoryUsageCounter.NativeMethods.GlobalMemoryStatusEx(MEMORYSTATUSEX& lpBuffer)
at Microsoft.Azure.Workflows.Common.PerformanceCounters.WebAppWorkerMemoryUsageCounter.ReadCounter()
at Microsoft.Azure.Workflows.Common.PerformanceCounters.FlowPerformance.get_CurrentMemoryUtilization()
at Microsoft.Azure.Workflows.Data.Configuration.EdgeResourceGovernancePolicy.<>c.<.ctor>b__21_0()
at Microsoft.Azure.Workflows.Data.Configuration.EdgeResourceGovernancePolicy.ShouldThrottleWorkerPulsation(JobTriggerEvent trigger, String& throttledTraceMessage)
at Microsoft.WindowsAzure.ResourceStack.Common.BackgroundJobs.Execution.JobDispatchingWorkerRegistry.CanAcquireDispatchingWorker(JobTriggerEvent trigger)
at Microsoft.WindowsAzure.ResourceStack.Common.BackgroundJobs.Notification.IBackgroundJobsNotificationChannelExtensions.CanAcquireDispatchingWorker(...)
at Microsoft.WindowsAzure.ResourceStack.Common.BackgroundJobs.Execution.JobExecutionEngine.PrepareJobDefinitionForImmediateExecution(...)
at Microsoft.WindowsAzure.ResourceStack.Common.BackgroundJobs.Execution.JobExecutionEngine.ScheduleJobExecution(...)
json
Symptoms
The failures follow a fixed cycle, regardless of what each test does. Across 33 tests in 4 test classes the results were fail fast → pass → pass → hang eight times in a row:
- Fail fast (2–3 s): the exception above, surfaced as "workflow is unhealthy".
- Pass (~3 s each): the next tests pass normally.
- Hang (~10 min): the next test never completes and fails with
Microsoft.Azure.Workflows.UnitTesting.Utilities.WaitTimeOutException: Exceeded maximum wait time. Message: '<null>'.
That cycle matches how FlowPerformance.CurrentMemoryUtilization works. When the cached sample has expired (Microsoft.WindowsAzure.Performance.MemorySamplingWindow, default 10 s), it moves the expiry forward before calling ReadCounter(), and nothing catches the exception. So on Linux roughly one call every 10 seconds throws, and every call in between gets the cached value:
- If the throw happens while a run is starting, the test fails fast.
- If it happens part-way through a run, while a later action is being dispatched in the background, the run never completes and the test hangs. This part is inferred from the stack trace and the timing; I haven't traced it directly.
Root cause (from inspecting the 1.0.3 assemblies)
-
FlowPerformance.GetMemoryUsageCounter()always returnsnew WebAppWorkerMemoryUsageCounter(), and that class P/Invokeskernel32!GlobalMemoryStatusEx. - By contrast,
FlowPerformance.GetProcessorUsageCounter()already checksIsContainerAppEnvironment(MANAGED_ENVIRONMENT/CONNECTED_ENVIRONMENT) and usesFlowProcessorUsageCounter.CreateLinuxContainerWorkerCounter()on Linux. The memory counter has no equivalent. - Memory-based throttling can't be turned off to avoid the call. In the embedded
AppSettings.resources,Microsoft.WindowsAzure.ResourceStack.JobResourceGovernance.MemoryBasedThrottlingEnabledis fixed atvalue="true", with noappSettingNameorhostSettingName. So neither app settings, environment variables norhost.jsoncan override it.
To reproduce
Reproduced on .NET 10 and .NET 8, in Microsoft's own SDK images:
- Create unit tests for a Standard workflow with the VS Code extension. Our test project uses
Microsoft.Azure.Workflows.WebJobs.Tests.Extension1.0.3 and MSTest 4.2.3, targetingnet10.0(also triednet8.0). - Build it and run
dotnet teston Windows: all 33 tests pass, on bothnet10.0andnet8.0. - Run the same compiled tests on Linux:
docker run --rm -v "<logic app workspace>:/src:ro" -w /src mcr.microsoft.com/dotnet/sdk:10.0 \
yaml
dotnet test Tests/<project>/bin/Release/net10.0/<project>.dll \
--results-directory /tmp/results --blame-hang-timeout 3m
The first test fails with the DllNotFoundException above, which the engine also logs as Workflow '<id>' validation and creation failed. Error: 'Unable to load shared library 'kernel32.dll'.... Two tests pass, then the fourth hangs until --blame-hang-timeout stops the run.
| Image | Runtime | Result |
|---|---|---|
| mcr.microsoft.com/dotnet/sdk:10.0 (Ubuntu 24.04.5, x86_64) | 10.0.14 | fail (4 s), pass, pass, hang |
| mcr.microsoft.com/dotnet/sdk:8.0 (Debian 12, x86_64) | 8.0.33 | fail (10 s), hang |
| Self-hosted Linux x64 GitHub Actions runner | 8.0.33 | fail, pass, pass, hang repeated 8 times; 16 of 33 failed |
Which tests pass, fail or hang depends on timing, because of the 10-second sample window. In the .NET 8 container the first test took 10 s, so the window ran out during the second test and that one hung.
Expected behaviour
Either:
- the unit tests run on Linux, so they can gate CI/CD on standard Linux build agents, or
- the documentation clearly states that the unit testing framework needs Windows, including when it runs in a pipeline.
Suggested fix
- Give the memory counter the same Linux path as the processor counter. On Linux, read memory from
/proc/meminfoor cgroup limits, or useGC.GetGCMemoryInfo(). At minimum, skip memory throttling when the counter isn't supported. - Catch exceptions from
ReadCounter()inCurrentMemoryUtilization, so a counter failure can't break or hang a workflow run. - Under the unit test host, consider not applying memory-based throttling at all.
Environment
-
Microsoft.Azure.Workflows.WebJobs.Tests.Extension1.0.3 (the latest on nuget.org at the time of writing) -
net10.0(runtime 10.0.14) andnet8.0(runtime 8.0.33) test projects - Fails on:
mcr.microsoft.com/dotnet/sdk:10.0(Ubuntu 24.04.5),mcr.microsoft.com/dotnet/sdk:8.0(Debian 12) and a self-hosted Linux x64 GitHub Actions runner - Works on: Windows 11,
net10.0andnet8.0
Workaround
Either run the unit tests on a Windows agent, or stop the engine from reading memory on Linux before any workflow runs.
The engine already has the path this needs. FlowPerformance.CurrentMemoryUtilization first checks MemoryUsageCounter, and if it's null it returns the cached sample without reading anything. Setting the counter to null on the shared FlowPerformance instance therefore removes every kernel32 call. We do this in an [AssemblyInitialize] in the test project:
using System;
using System.Reflection;
using Microsoft.VisualStudio.TestTools.UnitTesting;
[TestClass]
public static class LinuxWorkflowEngineWorkaround
{
private const string FlowPerformanceType =
"Microsoft.Azure.Workflows.Common.PerformanceCounters.FlowPerformance, Microsoft.Azure.Workflows.EventSource.Common";
[AssemblyInitialize]
public static void DisableWindowsOnlyMemoryCounter(TestContext context)
{
if (OperatingSystem.IsWindows())
{
return;
}
var type = Type.GetType(FlowPerformanceType, throwOnError: true)!;
var instance = type.GetProperty("Instance", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static)?.GetValue(null)
?? throw new InvalidOperationException("FlowPerformance.Instance not found; the Linux workaround needs updating.");
var counter = type.GetField("<MemoryUsageCounter>k__BackingField", BindingFlags.NonPublic | BindingFlags.Instance)
?? throw new InvalidOperationException("FlowPerformance.MemoryUsageCounter backing field not found; the Linux workaround needs updating.");
counter.SetValue(instance, null);
}
-
[AssemblyInitialize]runs once, before any test, so the counter is gone before the engine first checks memory. - It does nothing on Windows, where the counter works.
- On Linux it gets the shared instance through the static
FlowPerformance.Instanceand nulls the compiler-generated backing field of the get-onlyMemoryUsageCounterproperty. - It throws if any of those members are missing, so a package update fails loudly instead of quietly bringing the hangs back.
Results in mcr.microsoft.com/dotnet/sdk:10.0 (Ubuntu 24.04, runtime 10.0.14) with the workaround:
- All 33 tests pass in 1 m 43 s (about the same as on Windows), with no
DllNotFoundExceptionand no hangs. - Invalid definitions are still caught. With a workflow that adds an invalid action, the affected tests still fail in about 200 ms. So the workaround only removes the memory reading; it doesn't hide validation errors.
This relies on private internals, so it's only a stop-gap. It does suggest a small supported fix: have GetMemoryUsageCounter() return null, or a Linux counter, when not on Windows. CurrentMemoryUtilization already handles null.