Title: APIM Standard v2 behind Application Gateway: custom domain stopped working after Key Vault certificate rotation (502 Bad Gateway)
Environment
- Azure API Management Standard v2, accessed privately through a private endpoint
- Azure Application Gateway v2 (WAF) in front of APIM, with the backend pool pointing to the APIM private endpoint IP
- Custom gateway domain
api.contoso.com configured in APIM with a Key Vault certificate (unversioned secret identifier, auto-rotation)
- Public DNS:
api.contoso.com → A record pointing to the Application Gateway public IP
- During the initial deployment,
api.contoso.com was temporarily a CNAME to contoso-apim.azure-api.net, so the custom domain could be assigned. It was later changed to the A record so that all traffic goes through the WAF.
Symptom After a new certificate version was imported into Key Vault as part of a regular renewal, the Application Gateway started returning 502 Bad Gateway. Backend health showed:
The Common Name (CN) of the backend server certificate does not match the host header entered in the health probe configuration.
A TLS test against the private endpoint IP showed:
- SNI
api.contoso.com → the gateway presented CN=*.azurewebsites.net (platform default certificate)
- SNI
contoso-apim.azure-api.net → the gateway presented CN=*.azure-api.net and /status-0123456789abcdef returned 200
So the gateway had lost the custom domain binding, while the portal still showed the old certificate thumbprint in Custom domains.
What did not work
- Sync certificates: no effect. View sync logs showed no entries.
- Waiting for the automatic rotation. The Activity log showed
AUTO UPDATE SSL CERTIFICATE STARTED, and its payload already contained the new thumbprint, so Key Vault access and managed identity permissions were fine.
- Disabling the old certificate version in Key Vault.
- Updating the custom domain with a manually uploaded PFX: the operation
Create or Update API Management Service instance failed with a generic "Unable to Update API service at this time" error and the configuration rolled back.
Root cause During certificate rotation, Standard v2 appears to re-validate the custom domain against public DNS. That validation expects api.contoso.com to resolve (CNAME) to contoso-apim.azure-api.net. Because the public record points to the Application Gateway, the validation failed and the custom domain binding was dropped. The initial deployment worked only because the CNAME existed at the time the domain was assigned.
This behavior is not explicitly described in the official rotation documentation, which states that Key Vault certificates are picked up automatically without downtime in SLA tiers. It is, however, consistent with the Standard v2 limitation on publicly resolvable custom domain names, and with the recent Tech Community article "Your Certificate Renewed. Your Gateway Didn't Notice."
Resolution
- Temporarily changed the public DNS record
api.contoso.com from the A record to a CNAME → contoso-apim.azure-api.net (low TTL).
- After propagation, APIM validated the domain and applied the new certificate. Custom domains showed the new thumbprint, and the TLS test on SNI
api.contoso.com returned the correct certificate.
- Restored the public A record → Application Gateway public IP.
- Application Gateway backend health returned to Healthy (200) and the service was restored.
Lessons learned / recommendations
- With APIM v2 behind Application Gateway, Front Door or Traffic Manager, every certificate renewal may require the temporary CNAME step, unless the design changes.
- More robust alternatives:
- Configure the Application Gateway backend settings and probe to use the APIM default hostname (
contoso-apim.azure-api.net) and manage the public certificate only on the Application Gateway listener. This is the workaround suggested in the v2 documentation.
- Or use a different custom domain in APIM (for example
apim-internal.contoso.com) with a permanent public CNAME to *.azure-api.net.
- Keep a low TTL on the public record so the temporary change propagates quickly.
- Monitor the Activity log for
AUTO UPDATE SSL CERTIFICATE events and alert on failures.
Question for the community / Microsoft: Is there a supported way to validate domain ownership for a Standard v2 custom domain (for example, a TXT record) when the public DNS name must point to an Application Gateway or WAF, so that certificate rotation does not depend on changing the public DNS?Title: APIM Standard v2 behind Application Gateway: custom domain stopped working after Key Vault certificate rotation (502 Bad Gateway)
Environment
- Azure API Management Standard v2, accessed privately through a private endpoint
- Azure Application Gateway v2 (WAF) in front of APIM, with the backend pool pointing to the APIM private endpoint IP
- Custom gateway domain
api.contoso.com configured in APIM with a Key Vault certificate (unversioned secret identifier, auto-rotation)
- Public DNS:
api.contoso.com → A record pointing to the Application Gateway public IP
- During the initial deployment,
api.contoso.com was temporarily a CNAME to contoso-apim.azure-api.net, so the custom domain could be assigned. It was later changed to the A record so that all traffic goes through the WAF.
Symptom
After a new certificate version was imported into Key Vault as part of a regular renewal, the Application Gateway started returning 502 Bad Gateway. Backend health showed:
The Common Name (CN) of the backend server certificate does not match the host header entered in the health probe configuration.
A TLS test against the private endpoint IP showed:
- SNI
api.contoso.com → the gateway presented CN=*.azurewebsites.net (platform default certificate)
- SNI
contoso-apim.azure-api.net → the gateway presented CN=*.azure-api.net and /status-0123456789abcdef returned 200
So the gateway had lost the custom domain binding, while the portal still showed the old certificate thumbprint in Custom domains.
What did not work
- Sync certificates: no effect. View sync logs showed no entries.
- Waiting for the automatic rotation. The Activity log showed
AUTO UPDATE SSL CERTIFICATE STARTED, and its payload already contained the new thumbprint, so Key Vault access and managed identity permissions were fine.
- Disabling the old certificate version in Key Vault.
- Updating the custom domain with a manually uploaded PFX: the operation
Create or Update API Management Service instance failed with a generic "Unable to Update API service at this time" error and the configuration rolled back.
Root cause
During certificate rotation, Standard v2 appears to re-validate the custom domain against public DNS. That validation expects api.contoso.com to resolve (CNAME) to contoso-apim.azure-api.net. Because the public record points to the Application Gateway, the validation failed and the custom domain binding was dropped. The initial deployment worked only because the CNAME existed at the time the domain was assigned.
This behavior is not explicitly described in the official rotation documentation, which states that Key Vault certificates are picked up automatically without downtime in SLA tiers. It is, however, consistent with the Standard v2 limitation on publicly resolvable custom domain names, and with the recent Tech Community article "Your Certificate Renewed. Your Gateway Didn't Notice."
Resolution
- Temporarily changed the public DNS record
api.contoso.com from the A record to a CNAME → contoso-apim.azure-api.net (low TTL).
- After propagation, APIM validated the domain and applied the new certificate. Custom domains showed the new thumbprint, and the TLS test on SNI
api.contoso.com returned the correct certificate.
- Restored the public A record → Application Gateway public IP.
- Application Gateway backend health returned to Healthy (200) and the service was restored.
Lessons learned / recommendations
- With APIM v2 behind Application Gateway, Front Door or Traffic Manager, every certificate renewal may require the temporary CNAME step, unless the design changes.
- More robust alternatives:
- Configure the Application Gateway backend settings and probe to use the APIM default hostname (
contoso-apim.azure-api.net) and manage the public certificate only on the Application Gateway listener. This is the workaround suggested in the v2 documentation.
- Or use a different custom domain in APIM (for example
apim-internal.contoso.com) with a permanent public CNAME to *.azure-api.net.
- Keep a low TTL on the public record so the temporary change propagates quickly.
- Monitor the Activity log for
AUTO UPDATE SSL CERTIFICATE events and alert on failures.
Question for the community / Microsoft: Is there a supported way to validate domain ownership for a Standard v2 custom domain (for example, a TXT record) when the public DNS name must point to an Application Gateway or WAF, so that certificate rotation does not depend on changing the public DNS?