AKS cannot pull from private ABAC-enabled ACR: 401 persists after correct kubelet roles, reconcile, node-image refresh, and pull secret

Gregory DuPont 0 Reputation points
2026-10-27T22:03:51.2166667+00:00

I have a private AKS cluster in Central US and a private Azure Container Registry with public network access disabled. The registry uses “RBAC Registry + ABAC Repository Permissions.”

A one-time Job cannot pull this immutable image:

<private-registry>.azurecr.io/estate-planning/worker@sha256:4e3de791d211ae13dc1ea387d095518aaca321b0df1c7ba00d60b13d24fba187

The Job schedules successfully but never starts its container. Pod events repeatedly show:

failed to pull and unpack image ... failed to resolve image ...

failed to authorize: failed to fetch anonymous token:

unexpected status from GET request to

https://<private-registry>.azurecr.io/oauth2/token?scope=repository%3Aestate-planning%2Fworker%3Apull&service=<private-registry>.azurecr.io:

401 Unauthorized

Configuration verified:

  • AKS uses a user-assigned kubelet managed identity.
  • That exact kubelet identity is attached to the system VM scale set.
  • The identity has:
    • Container Registry Repository Reader, restricted by ABAC condition to estate-planning/worker
    • Container Registry Repository Catalog Lister
  • The ACR private endpoint is approved and private DNS is linked.
  • The image is built as linux/amd64; the AKS Linux system pool uses Standard_D2s_v6.
  • The verification service account has no pre-existing imagePullSecret.

Troubleshooting already completed:

  1. Reconciled AKS after confirming the ABAC role assignments.
  2. Performed a node-image-only upgrade of the existing system pool.
  3. Retried the pull after each step; the same 401 occurred.
  4. Created a temporary ACR token restricted to estate-planning/worker with only content/read and metadata/read.
  5. Created a temporary kubernetes.io/dockerconfigjson imagePullSecret and referenced it only in one retry Job.
  6. That retry still reported the same anonymous-token 401.
  7. All retry Jobs, temporary secrets, tokens, scope maps, and temporary namespace Writer permissions were deleted afterward.

No client or production data was accessed, and no persistent workload was deployed.

Questions:

  1. Is this a known AKS + private ABAC-enabled ACR authentication issue?
  2. Why does AKS report an anonymous-token request even when the pod specifies a valid repository-scoped imagePullSecret?
  3. What supported, least-privilege diagnostic can verify whether this digest is retrievable from the private registry without introducing a VM or broad registry access?
  4. Are there known constraints for AKS pulling by immutable digest from an ABAC-enabled private ACR?
Azure Kubernetes Service
Azure Kubernetes Service

An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.

0 comments

6 answers

Sort by: Newest
Most helpful Newest Oldest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    Comments have been turned off. Learn more

  2. Gregory DuPont 0 Reputation points
    2026-10-28T13:11:58.0833333+00:00

    Thank you. We completed the recommended controlled ACR diagnostic retry after enabling ContainerRegistryLoginEvents and ContainerRegistryRepositoryEvents.

    Results:

    • The retry Job again failed before container start with ImagePullBackOff and failed to fetch anonymous token: 401.
    • ACR login events show the AKS kubelet managed identity successfully authenticated to the registry with 200.
    • About 0.1 seconds later, the same AKS node runtime (containerd/2.3.7-3) made an unauthenticated request that ACR rejected with 401.
    • No repository events were recorded for estate-planning/worker, so the request failed before manifest or layer retrieval.
    • The earlier temporary repository-scoped ACR token and Kubernetes imagePullSecret retry produced the same outcome. The secret was referenced only by that retry Job and was deleted afterward.

    Therefore, connectivity, private DNS, private endpoint access, and successful kubelet managed-identity authentication have been demonstrated. The unresolved issue is why the AKS/containerd pull flow proceeds to an unauthenticated request after the managed-identity login succeeds, without reaching repository authorization.

    Can you confirm whether this sequence is expected for AKS containerd/2.3.7-3 with an ABAC-enabled private ACR, and identify the supported remediation or escalation path? We can provide redacted UTC timestamps and correlation IDs privately if required.

    Was this answer helpful?

    0 comments
  3. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    Comments have been turned off. Learn more

  4. Gregory DuPont 0 Reputation points
    2026-10-28T12:59:43.42+00:00

    Thank you. We completed the recommended controlled ACR diagnostic retry after enabling ContainerRegistryLoginEvents and ContainerRegistryRepositoryEvents.

    Results:

    • The retry Job again failed before container start with ImagePullBackOff and failed to fetch anonymous token: 401.
    • ACR login events show the AKS kubelet managed identity successfully authenticated to the registry with 200.
    • About 0.1 seconds later, the same AKS node runtime (containerd/2.3.7-3) made an unauthenticated request that ACR rejected with 401.
    • No repository events were recorded for estate-planning/worker, so the request failed before manifest or layer retrieval.
    • The earlier temporary repository-scoped ACR token and Kubernetes imagePullSecret retry produced the same outcome. The secret was referenced only by that retry Job and was deleted afterward.

    Therefore, connectivity, private DNS, private endpoint access, and successful kubelet managed-identity authentication have been demonstrated. The unresolved issue is why the AKS/containerd pull flow proceeds to an unauthenticated request after the managed-identity login succeeds, without reaching repository authorization.

    Can you confirm whether this sequence is expected for AKS containerd/2.3.7-3 with an ABAC-enabled private ACR, and identify the supported remediation or escalation path? We can provide redacted UTC timestamps and correlation IDs privately if required.

    Was this answer helpful?

    0 comments
  5. Gregory DuPont 0 Reputation points
    2026-10-28T12:55:33.2533333+00:00

    Update: controlled ACR diagnostic result

    We enabled temporary ACR login and repository-event diagnostics, then created exactly one closed-host retry Job using the same immutable Worker image digest. The Job again failed before the container started with the same ImagePullBackOff/ anonymous-token 401.

    The ACR login events now show the following sequence from the AKS node’s containerd/2.3.7-3 runtime:

    1. The configured AKS kubelet managed identity authenticated successfully to the registry (200).
    2. Roughly 0.1 seconds later, the same runtime made an unauthenticated login/token request, which ACR rejected (401).
    3. No ContainerRegistryRepositoryEvents were recorded for estate-planning/worker.

    This indicates the request reaches ACR and the kubelet identity itself is accepted. The failure occurs after successful managed-identity authentication but before any repository manifest or layer read.

    We removed the temporary Job, diagnostic setting, and namespace-scoped Writer assignment afterward.

    Could you advise why AKS containerd falls back to an unauthenticated token request after the managed-identity login succeeds in a private ACR using ABAC repository permissions? Is there a supported AKS/ACR configuration or known issue affecting this credential handoff?

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.