An Apache Spark-based analytics platform optimized for Azure.
Azure Databricks managed VNet: allow classic compute to a firewall-enabled ADLS Gen2 account in the same region (service endpoint on workers-vnet blocked by deny assignment)
Hello,
We run an Azure Databricks workspace in West Europe that uses the Databricks-managed network (no VNet injection). Classic job compute (DBR 17.3 LTS, Photon, standard access mode) needs to write to an ADLS Gen2 storage account in the same region through a Unity Catalog volume. The storage account firewall has default action Deny, bypass None, and a resource instance rule for the Unity Catalog access connector.
Problem:
Writes fail with HTTP 403, error code AuthorizationFailure ("This request is not authorized to perform this operation"), for example from dbutils.fs.put to a UC volume path. IP rules do not help because the storage account and the workspace are in the same region, and the managed workers-vnet egresses through a NAT gateway.
What we tried:
Adding the Microsoft.Storage service endpoint to public-subnet and private-subnet of workers-vnet (in the managed resource group) so we can add a virtual network rule on the storage account. Saving the subnet fails with: the client has permission for Microsoft.Network/virtualNetworks/subnets/write, but does not have permission to perform 'Microsoft.Network/natGateways/join/action' on the nat-gateway because it is blocked by deny assignments.
Constraint: we do not have a paid Azure support plan, so we cannot open a technical support ticket.
Questions:
- Is adding a Microsoft.Storage service endpoint to the managed VNet subnets a supported configuration? If so, how can the deny assignment be handled without Azure Support?
- Which approach would you recommend instead: converting the workspace to VNet injection (preview), a private endpoint from workers-vnet to the storage account, or something else? Any pitfalls for each?
- Does a resource instance rule for the access connector ever cover classic compute that uses a Unity Catalog vended SAS, or does the compute network always have to be allowed on the storage firewall separately?
Thank you!