AKS cannot pull from private ABAC-enabled ACR: 401 persists after correct kubelet roles, reconcile, node-image refresh, and pull secret

Gregory DuPont 0 Reputation points
2026-10-27T22:03:51.2166667+00:00

I have a private AKS cluster in Central US and a private Azure Container Registry with public network access disabled. The registry uses “RBAC Registry + ABAC Repository Permissions.”

A one-time Job cannot pull this immutable image:

<private-registry>.azurecr.io/estate-planning/worker@sha256:4e3de791d211ae13dc1ea387d095518aaca321b0df1c7ba00d60b13d24fba187

The Job schedules successfully but never starts its container. Pod events repeatedly show:

failed to pull and unpack image ... failed to resolve image ...

failed to authorize: failed to fetch anonymous token:

unexpected status from GET request to

https://<private-registry>.azurecr.io/oauth2/token?scope=repository%3Aestate-planning%2Fworker%3Apull&service=<private-registry>.azurecr.io:

401 Unauthorized

Configuration verified:

  • AKS uses a user-assigned kubelet managed identity.
  • That exact kubelet identity is attached to the system VM scale set.
  • The identity has:
    • Container Registry Repository Reader, restricted by ABAC condition to estate-planning/worker
    • Container Registry Repository Catalog Lister
  • The ACR private endpoint is approved and private DNS is linked.
  • The image is built as linux/amd64; the AKS Linux system pool uses Standard_D2s_v6.
  • The verification service account has no pre-existing imagePullSecret.

Troubleshooting already completed:

  1. Reconciled AKS after confirming the ABAC role assignments.
  2. Performed a node-image-only upgrade of the existing system pool.
  3. Retried the pull after each step; the same 401 occurred.
  4. Created a temporary ACR token restricted to estate-planning/worker with only content/read and metadata/read.
  5. Created a temporary kubernetes.io/dockerconfigjson imagePullSecret and referenced it only in one retry Job.
  6. That retry still reported the same anonymous-token 401.
  7. All retry Jobs, temporary secrets, tokens, scope maps, and temporary namespace Writer permissions were deleted afterward.

No client or production data was accessed, and no persistent workload was deployed.

Questions:

  1. Is this a known AKS + private ABAC-enabled ACR authentication issue?
  2. Why does AKS report an anonymous-token request even when the pod specifies a valid repository-scoped imagePullSecret?
  3. What supported, least-privilege diagnostic can verify whether this digest is retrievable from the private registry without introducing a VM or broad registry access?
  4. Are there known constraints for AKS pulling by immutable digest from an ABAC-enabled private ACR?
Azure Kubernetes Service
Azure Kubernetes Service

An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.

0 comments

6 answers

Sort by: Newest
Most helpful Newest Oldest
  1. Jose Benjamin Solis Nolasco 12,941 Reputation points Volunteer Moderator
    2026-10-27T22:19:36.6433333+00:00

    Welcome to Microsoft Q&A

    Hello @Gregory DuPont I hope you are doing well,

    I;m thinking , this does not appear to be a typical AKS-to-ACR connectivity issue. You've already validated the kubelet identity, ABAC role assignments, private endpoint connectivity, DNS resolution, reconciliation, and node image refresh.

    A few observations:

    • There is currently no documented limitation that prevents AKS from pulling an image by immutable digest from an ABAC-enabled ACR.
    • The message: failed to fetch anonymous token does not necessarily mean AKS intentionally attempted an anonymous pull. It indicates that the registry authentication flow reached a point where ACR returned a 401 during token acquisition. The kubelet event alone doesn't conclusively identify which credential path was evaluated.
      • Since both the managed identity approach and the repository-scoped token/imagePullSecret approach produce the same result, the next step is to determine what ACR actually received rather than relying solely on the pod event text.

    I would recommend reviewing ACR diagnostic logs if they are available:

    ContainerRegistryLoginEvents

    | where TimeGenerated > ago(2h)

    | project TimeGenerated, Identity, ResultDescription

    | order by TimeGenerated desc

    and:

    ContainerRegistryRepositoryEvents

    | where Repository == "estate-planning/worker"

    | order by TimeGenerated desc

    These logs can help determine:

    • Whether the request reached ACR.
    • Which identity was presented.
    • Whether the ABAC condition evaluation failed.
    • Whether the repository authorization succeeded but manifest retrieval failed.

    Also, verify the live pod definition rather than the deployment manifest to confirm the pull secret is actually attached and matches the exact login server hostname:

    kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets[*].name}'

    and decode the stored .dockerconfigjson to verify the registry server entry exactly matches:

    <private-registry>.azurecr.io

    One additional data point that may help: were ACR diagnostic logs enabled before the failed pull attempts? If not, enabling them and performing a single controlled retry may provide the clearest evidence of whether the failure is occurring during authentication, ABAC authorization, or manifest retrieval.

    Useful references:

    If ACR logs show the correct identity, repository (estate-planning/worker), and successful role evaluation but the registry still returns 401, I would recommend opening a Microsoft support case with the AKS resource ID, ACR resource ID, UTC timestamps, and relevant ACR login/repository log entries, as that may indicate a service-side issue requiring backend investigation.

    If my answer helped you resolve your issue, please consider marking it as the correct answer. This helps others in the community find similar solutions.

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.