Azure Local solution updates blocked by malformed EnvironmentReadiness Timestamp

Juhani Harju 0 Reputation points
2026-10-20T12:20:41.2433333+00:00

Problem description

We have a three-node Azure Local 24H2 cluster where the system health and update readiness results cannot be deserialized.

The latest fresh SystemHealth run failed with:

Newtonsoft.Json.JsonReaderException Could not convert string to DateTime: 2026-10-15T09.58.03.629Z

Expected ISO 8601 format: 2026-10-15T09:58:03.629Z

The latest failure occurred at:

[72].Timestamp, line 2516, position 48

Previous fresh runs failed with different malformed timestamps and at

different result indexes, including [64].Timestamp. This confirms that

the issue is reproducible and is not limited to a single stale cached

result.

Because deserialization of the complete readiness result array fails,

Get-SolutionUpdateEnvironment returns HealthState=Error and only a

synthetic CRITICAL parsing error. The underlying health check results,

their sources, target resources, and remediation information cannot be

displayed.

Environment:

Azure Local version: 12.2609.1003.7

Azure Local OS version: 24H2

OS build: 26100.33438

Region: West Europe

Hardware partner: HPE

Cluster size: Three nodes

HPE Minimal SBE: 4.2.2606.1

AzStackHci.EnvironmentChecker: 10.2609.0.2002

Microsoft.AzureStack.LcmUpdateService.PowerShell: 2.2608.0.2

We have completed the following troubleshooting:

  • Verified that all three nodes have synchronized clocks.
  • Verified that W32Time is running on all nodes.
  • Verified the FLE Standard Time zone and the correct UTC+3 offset.
  • Verified that the nodes use valid domain controller time sources.
  • Changed and verified the effective NT AUTHORITY\SYSTEM culture as en-US on all three nodes.
  • Copied the regional settings to system accounts.
  • Restarted all three nodes individually.
  • Reproduced the issue with newly generated timestamps after the restarts.
  • Ran Get-SolutionUpdateEnvironment -FullHealthCheckDetails. It fails before the underlying results can be displayed.
  • Ran Microsoft.AzLocal.CSSTools. The latest analysis reports zero failures.
  • Submitted Azure Local diagnostic logs through Diagnostics and Remote Support.
  • Verified that Get-SolutionUpdate currently returns no available update resources, so a PrepareOnly run cannot be performed.
  • Inspected the installed and staged HPE SBE content. No source code using the malformed HH.mm.ss timestamp format was found.
  • Verified that the HPE SBE returns Timestamp as a valid UTC System.DateTime value.
  • Verified that the Microsoft EnvironmentReadiness result object preserves Timestamp as System.DateTime with DateTime.Kind=Utc.
  • The malformed period-separated value appears to be introduced later by EnvironmentReadiness or Update Service processing.

Current status

The supported Azure Local solution update readiness workflow remains

blocked. The cluster is intended for production VM, Kubernetes, and

potentially VDI workloads.

Please escalate this issue to the Azure Local Update Service /

EnvironmentReadiness engineering team and provide a supported

workaround, hotfix, or remediation plan.

Please use the linked support request or a private message for

customer-specific identifiers, logs, and remote-access information.

Azure Local
0 comments No comments

2 answers

Sort by: Most helpful
  1. Alex Burlachenko 26,695 Reputation points MVP Volunteer Moderator
    2026-10-23T14:10:16.3166667+00:00

    Hi Juhani Harju & thx for join me at Q&A platform,

    an EnvironmentReadiness / Update Service serialization issue, rather than an actual cluster health or time synchronization problem.

    The important clue is the timestamp format: 2026-10-15T09.58.03.629Z. The date and UTC suffix are valid, but the time portion uses periods instead of colons. Newtonsoft.Json then fails to deserialize the result, which prevents Get-SolutionUpdateEnvironment from displaying the actual readiness checks.

    There's an interesting .NET detail here. When formatting a DateTime, the : character in a custom format string can act as a culture-dependent time separator. Depending on the effective culture, something like HH:mm:ss can produce 09.58.03 instead of 09:58:03. That matches the malformed value you're seeing.

    It doesn't prove that's the bug, tho. You've already changed the SYSTEM culture to en-US and reproduced the issue after restarting all nodes. But the component generating the JSON might be using a different execution context, an explicitly selected culture, or its own formatting logic.

    The fact that the HPE SBE and Microsoft EnvironmentReadiness objects contain valid UTC System.DateTime values is particularly useful. It suggests the issue occurs during serialization or at another processing boundary, rather than when the original health checks generate their timestamps.

    At this point, I wouldn't spend more time changing regional settings or restarting nodes. You've already done enough testing to rule out the usual configuration issues.

    What Microsoft engineering needs to identify is which component first converts the valid DateTime into the malformed string. I'd ask them to inspect the raw readiness payload at the EnvironmentReadiness-to-Update Service boundary, confirm the runtime culture used by the serializer, and check whether the affected component uses a culture-sensitive timestamp format instead of an invariant ISO 8601 format.

    A proper fix should serialize these values consistently, for example using the invariant round-trip O format, rather than relying on machine or process culture.

    I agree this needs escalation to the Azure Local Update Service / EnvironmentReadiness engineering team. The changing result indexes, fresh reproductions, and successful validation of the original HPE timestamps make a strong case for a product-side defect.

    I wouldn't recommend manually modifying readiness JSON, bypassing the health checks, or forcing a solution update on a production cluster. Those aren't supported fixes and could hide genuine readiness failures.

    One thing worth including in the existing support case is the exact component responsible for producing the malformed JSON, if Microsoft can identify it from the diagnostic logs, along with the EnvironmentChecker and LcmUpdateService versions you've provided. That should help engineering determine whether this is a known serialization regression in the 2609 update stack.

    Given how thoroughly you've isolated the issue, the next useful response really needs to come from Microsoft engineering a confirmed root cause, supported workaround, or hotfix.

    rgds,

    Alex

    p.s. if my answer is help - pls mark it as an answer

    Was this answer helpful?

    0 comments No comments

  2. Saraswathi Devadula 16,040 Reputation points Microsoft External Staff Moderator
    2026-10-23T03:43:50.7966667+00:00

    Hello Juhani Harju

    Kindly please change the timestamp assignment in MemoryResourceSummaryHelper.psm1 to $resultObject.TimeStamp = Get-Now, replacing the current Get-Date -Format expression. The file is in the active Microsoft.EnvironmentReadiness.Validator module under %ProgramFiles%\WindowsPowerShell\Modules\Microsoft.EnvironmentReadiness.Validator. This keeps the memory check enabled while using the shared timestamp helper instead of the culture-sensitive formatter.

    Then rerun Invoke-SolutionUpdatePrecheck -SystemHealth and share the full output so we can confirm whether the timestamp parsing error has cleared.

    Will this change be included in a future Microsoft.EnvironmentReadiness.Validator release, and should we keep the workaround in place until then?

    Yes, this will come in future environment readiness release. You need to keep the existing workaround in place as per our previous instructions

    Was this answer helpful?

    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.