An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
AKS Automatic 1.36.5: MutatingWebhookConfiguration denied by aks-managed-require-webhook-protected-namespace-exclusion, matchCondition not auto-injected
On an AKS Automatic cluster running Kubernetes 1.36.5, I can't create any MutatingWebhookConfiguration unless I add an AKS-specific matchCondition by hand. The denial message says AKS should inject this condition automatically on 1.36+, but that isn't happening.
Environment
- AKS Automatic, Kubernetes 1.36.5 (control plane and nodes)
- Region: East US
- Node image: AKSAzureLinux-V3gen2fips-202610.30.1
- Azure Policy add-on enabled. It is not the cause: the denial comes from a native ValidatingAdmissionPolicy, not from Gatekeeper.
What happens
Installing the CloudNativePG Helm chart (cloudnative-pg 0.29.1) fails with:
mutatingwebhookconfigurations.admissionregistration.k8s.io "cnpg-mutating-webhook-configuration"
is forbidden: ValidatingAdmissionPolicy 'aks-managed-require-webhook-protected-namespace-exclusion'
with binding 'aks-managed-require-webhook-protected-namespace-exclusion-binding' denied request:
Every webhook in a MutatingWebhookConfiguration must carry the matchCondition
'aks-managed-exclude-system-namespaces', which prevents the webhook from mutating AKS-managed
workloads in protected namespaces such as kube-system. AKS injects it automatically on Kubernetes
1.36.0 and later, so this error on those clusters means the injection did not run: contact support.
Reproduce
kubectl apply --dry-run=server -f <any MutatingWebhookConfiguration> is denied.
The same object is accepted once each webhook has:
matchConditions:
- name: aks-managed-exclude-system-namespaces
expression: '!(has(request.namespace) && request.namespace in ["kube-system", "aks-istio-system", "aks-static-egress-gateway", "azuresecuritylinuxagent", "aks-istio-egress", "aks-istio-ingress", "fleet-system", "acstor", "azappconfig-system", "azure-extensions-usage-system", "dataprotection-microsoft", "gatekeeper-system", "sc-system"])'
The policy checks that the expression matches its requiredExpression variable exactly, so you need to copy it character for character.
What I checked
- There are no
MutatingAdmissionPolicyorMutatingAdmissionPolicyBindingresources in the cluster. - None of the existing mutating webhooks have rules for
admissionregistration.k8s.io/mutatingwebhookconfigurations. The ones present areaks-webhook-admission-controller,aks-node-mutating-webhook,safeguards-workload-mutating-webhook,azure-wi-webhook-mutating-webhook-configuration,gatekeeper-mutating-webhook-configurationandvpa-webhook-config. - So no injector appears to be deployed.
Workaround
I add the condition through a Helm post-renderer. With Flux it looks like this:
spec:
postRenderers:
- kustomize:
patches:
- target:
kind: MutatingWebhookConfiguration
name: cnpg-mutating-webhook-configuration
patch: |
- op: add
path: /webhooks/0/matchConditions
value: &aks-exclusion
- name: aks-managed-exclude-system-namespaces
expression: '<expression above>'
- op: add
path: /webhooks/1/matchConditions
value: *aks-exclusion
# ...repeat for each webhook in the configuration
Questions
- Is anyone else seeing this on AKS Automatic 1.36.x? Is the injector rolled out separately from the policy?
- Once injection works, does it leave alone a webhook that already has the condition, or will it add a duplicate? I want to know whether the workaround needs to be removed first.
- Will the protected-namespace list in
requiredExpressionchange over time? Because the policy requires an exact match, a hand-added copy would start failing whenever the list changes.