Storage 403 "Request date header too old" while the server's Time was about 44 minutes ahead (Nov 14, 2026)

BradBeverly-5832 0 Reputation points
2026-11-16T15:21:54.8766667+00:00

## Problem description

On November 14, 2026, from about 13:11 to 22:00 UTC, a storage account in Central US intermittently rejected Shared Key requests with HTTP 403 AuthenticationFailed ("Server failed to authenticate the request"). Other requests from the same clients succeeded at the same time. It has not happened since, and it did not happen in the 30 days before. The failures delayed queue processing in our Functions app and failed some API requests in our web app.

Our telemetry recorded 620 failures: 551 on the Queue service and 69 on the Table service. 551 of them include AuthenticationErrorDetail: Request date header too old; the others do not include a detail field.

In every failure, the server Time in the error body is ahead of when the request was sent: by about 44 minutes 38 seconds in 604 of the 620, and by 33 to 68 minutes in the other 16. Example: a request sent with date Sat, 14 Nov 2026 14:05:15 GMT came back with server Time 2026-11-14T14:49:54Z.

Shared Key rejects a request whose date is more than 15 minutes from the server's time. A server clock running about 45 minutes fast would reject every request it handled as too old, which matches what we saw.

## Environment

- General-purpose v2 storage account, Standard_LRS, Central US

- The failing requests use Shared Key (connection string with the account key)

- Public network access from all networks; no firewall rules, no private endpoints

- Clients: an Azure Functions app (Consumption plan, .NET isolated worker) and an Azure App Service web app, both using the Azure SDK for .NET on Azure-managed hosts

## What I've already tried

- Activity log: no key regeneration and no configuration, network or role changes in the 30 days before. The same keys worked before, during and after the window, and the failures stopped on their own.

- Client clocks: the same offset shows up on 71 different Functions instances and on the separate App Service. Separate hosts can't all be off by the same amount, and our timestamps agreed with outside systems during the window.

- Azure Monitor: the account's Availability and Transactions metrics (split by response type) look normal for that day and record no authentication errors. Resource Health shows Available throughout, and Service Health has no Storage event for our subscription.

- The portal's "Diagnostic for failed authentication requests", run with one of the failed request IDs, returned only an unrelated 409 Conflict template.

- A support case is open on our Standard plan; it now requires this post to continue.

## Questions

1. Did the Storage front ends serving this account have a clock or time sync problem on 2026-11-14 between 13:11 and 22:00 UTC?

2. If so, what was the root cause, and is there a post-incident report?

3. Why don't requests rejected this way show up in the account's Transactions metric or in Resource Health?

The storage account name, failed request IDs and the support case number are available to Microsoft staff by private message.

Azure Storage
Azure Storage

Globally unique resources that provide access to data management services and serve as the parent namespace for the services.

0 comments No comments

1 answer

Sort by: Newest
  1. Pavan Addepalli 0 Reputation points
    2026-11-19T11:27:33.7066667+00:00

    Your evidence warrants a Storage service-side investigation. However, the timestamp discrepancy alone does not confirm which component had an incorrect clock or establish a root cause.

    1. Was there a Storage clock problem?

    Microsoft documents that Shared Key requests must be no older than 15 minutes when they reach Storage. If both Date and x-ms-date are present, x-ms-date takes precedence. Your reported offsets are therefore sufficient to explain the explicit “Request date header too old” rejections.

    Assuming the recorded per-attempt send and receive timestamps show a short round trip, and the actual outgoing date header matches UTC, the repeated offset across independent clients is strong evidence consistent with a service-side time discrepancy. It is not, by itself, proof that particular Storage front ends were running fast. The 69 failures without an authentication-detail field also cannot conclusively be assigned the same cause.

    1. How can the root cause and incident report be obtained?

    Since you already have a support case, request a Storage engineering investigation, rather than repeating key, firewall, or RBAC troubleshooting.

    Provide the following through the existing case’s secure channel: representative failed x-ms-request-id values from both services, nearby successful request IDs, actual request date headers, client send/receive timestamps, response timestamps and bodies, and SDK versions.

    Ask support to correlate these with service-side telemetry, investigate the repeated time offset, and provide any associated incident tracking ID, customer-facing RCA, or post-incident report. A public Q&A response cannot independently establish those internal findings.

    1. Why were metrics and Resource Health normal?

    These are separate questions:

    • Transactions: Microsoft documents successful and failed requests in this metric, including a ResponseType=AuthenticationError category for Queue and Table Storage. Consequently, it would be incorrect to claim that authentication failures are universally excluded. The missing entries need reconciliation by support against the failed request IDs and the service-specific metric scopes.

    • Availability: This metric uses billable requests and applicable requests, including unexpected errors; it is not simply your application’s percentage of successful requests. A normal value does not establish that no authentication failures occurred.

    • Resource Health: Its documented Storage checks assess platform-related read/write failures and cluster availability. Inferring from that scope, an Available status does not independently rule out this particular intermittent authentication problem. The reason it did not detect your event still requires investigation.

    1. Should clients retry this particular 403?

    Do not retry every 403 indiscriminately. Azure.Core’s default retry classifier does not classify HTTP 403 as retryable, and the Storage-specific classifier does not add AuthenticationFailed to its retryable error codes. Increasing the retry count alone therefore does not address this classification.

    As an application-specific mitigation—not a documented fix for this incident—consider a small, bounded retry with exponential backoff and jitter, restricted to 403 / AuthenticationFailed with the explicit stale-date detail. Reissue through the SDK so the request is freshly dated and signed. Preserve failure telemetry, respect the operation’s timeout, and account for idempotency and existing Functions/SDK retries. Microsoft’s retry guidance emphasizes these safeguards.

    This may help if the fault is intermittent, but it does not repair an incorrect service clock. The incident’s cause, monitoring gap, and any incident-specific mitigation remain unverified pending the support investigation.

    Was this answer helpful?

    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.