Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
Your evidence warrants a Storage service-side investigation. However, the timestamp discrepancy alone does not confirm which component had an incorrect clock or establish a root cause.
- Was there a Storage clock problem?
Microsoft documents that Shared Key requests must be no older than 15 minutes when they reach Storage. If both Date and x-ms-date are present, x-ms-date takes precedence. Your reported offsets are therefore sufficient to explain the explicit “Request date header too old” rejections.
Assuming the recorded per-attempt send and receive timestamps show a short round trip, and the actual outgoing date header matches UTC, the repeated offset across independent clients is strong evidence consistent with a service-side time discrepancy. It is not, by itself, proof that particular Storage front ends were running fast. The 69 failures without an authentication-detail field also cannot conclusively be assigned the same cause.
- How can the root cause and incident report be obtained?
Since you already have a support case, request a Storage engineering investigation, rather than repeating key, firewall, or RBAC troubleshooting.
Provide the following through the existing case’s secure channel: representative failed x-ms-request-id values from both services, nearby successful request IDs, actual request date headers, client send/receive timestamps, response timestamps and bodies, and SDK versions.
Ask support to correlate these with service-side telemetry, investigate the repeated time offset, and provide any associated incident tracking ID, customer-facing RCA, or post-incident report. A public Q&A response cannot independently establish those internal findings.
- Why were metrics and Resource Health normal?
These are separate questions:
• Transactions: Microsoft documents successful and failed requests in this metric, including a ResponseType=AuthenticationError category for Queue and Table Storage. Consequently, it would be incorrect to claim that authentication failures are universally excluded. The missing entries need reconciliation by support against the failed request IDs and the service-specific metric scopes.
• Availability: This metric uses billable requests and applicable requests, including unexpected errors; it is not simply your application’s percentage of successful requests. A normal value does not establish that no authentication failures occurred.
• Resource Health: Its documented Storage checks assess platform-related read/write failures and cluster availability. Inferring from that scope, an Available status does not independently rule out this particular intermittent authentication problem. The reason it did not detect your event still requires investigation.
- Should clients retry this particular 403?
Do not retry every 403 indiscriminately. Azure.Core’s default retry classifier does not classify HTTP 403 as retryable, and the Storage-specific classifier does not add AuthenticationFailed to its retryable error codes. Increasing the retry count alone therefore does not address this classification.
As an application-specific mitigation—not a documented fix for this incident—consider a small, bounded retry with exponential backoff and jitter, restricted to 403 / AuthenticationFailed with the explicit stale-date detail. Reissue through the SDK so the request is freshly dated and signed. Preserve failure telemetry, respect the operation’s timeout, and account for idempotency and existing Functions/SDK retries. Microsoft’s retry guidance emphasizes these safeguards.
This may help if the fault is intermittent, but it does not repair an incorrect service clock. The incident’s cause, monitoring gap, and any incident-specific mitigation remain unverified pending the support investigation.