Azure Databricks managed VNet: allow classic compute to a firewall-enabled ADLS Gen2 account in the same region (service endpoint on workers-vnet blocked by deny assignment)

Question

Azure Databricks managed VNet: allow classic compute to a firewall-enabled ADLS Gen2 account in the same region (service endpoint on workers-vnet blocked by deny assignment)

Jerrel den Harder 45

Hello,

We run an Azure Databricks workspace in West Europe that uses the Databricks-managed network (no VNet injection). Classic job compute (DBR 17.3 LTS, Photon, standard access mode) needs to write to an ADLS Gen2 storage account in the same region through a Unity Catalog volume. The storage account firewall has default action Deny, bypass None, and a resource instance rule for the Unity Catalog access connector.

Problem:

Writes fail with HTTP 403, error code AuthorizationFailure ("This request is not authorized to perform this operation"), for example from dbutils.fs.put to a UC volume path. IP rules do not help because the storage account and the workspace are in the same region, and the managed workers-vnet egresses through a NAT gateway.

What we tried:

Adding the Microsoft.Storage service endpoint to public-subnet and private-subnet of workers-vnet (in the managed resource group) so we can add a virtual network rule on the storage account. Saving the subnet fails with: the client has permission for Microsoft.Network/virtualNetworks/subnets/write, but does not have permission to perform 'Microsoft.Network/natGateways/join/action' on the nat-gateway because it is blocked by deny assignments.

Constraint: we do not have a paid Azure support plan, so we cannot open a technical support ticket.

Questions:

Is adding a Microsoft.Storage service endpoint to the managed VNet subnets a supported configuration? If so, how can the deny assignment be handled without Azure Support?
Which approach would you recommend instead: converting the workspace to VNet injection (preview), a private endpoint from workers-vnet to the storage account, or something else? Any pitfalls for each?
Does a resource instance rule for the access connector ever cover classic compute that uses a Unity Catalog vended SAS, or does the compute network always have to be allowed on the storage firewall separately?

Thank you!

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Hello Jerrel den Harder

The behavior you're seeing is expected with a Databricks-managed VNet, and the 403 AuthorizationFailure is consistent with the storage firewall blocking classic compute traffic.

1. Can you enable service endpoints on workers-vnet?

Not through a supported customer-managed configuration.

The Databricks-managed resource group is protected by system deny assignments. Even Owner permissions cannot bypass them.

I see confirmed behavior in a similar Q&A case involving the Databricks NAT Gateway and subnet deny assignments.

I would not recommend attempting to remove the deny assignment or modify the managed VNet directly.

2. What are the supported alternatives?

Option A - Migrate to VNet injection (recommended for classic compute)

Here is in-place migration from a managed VNet to a customer-managed VNet:

Update workspace network configuration

After migration, you can configure:

Microsoft.Storage service endpoints and storage VNet rules.
Private endpoints for ADLS Gen2 (dfs and blob), with the required private DNS configuration.

Network-update capability is in Public Preview, requires stopping running compute and jobs, and does not currently support Terraform for the migration.

Option B - Serverless compute with NCC

If your workload supports serverless jobs, you can configure private connectivity through a Network Connectivity Configuration (NCC), without changing the workspace VNet.

Configure private connectivity to Azure resources.

This requires the Databricks Premium plan and applies to serverless compute only, not your existing classic job clusters.

Option C - Private endpoint in workers-vnet

I would not recommend this approach because creating and configuring private endpoints and DNS inside the Databricks-managed VNet encounters the same management restrictions.

3. Does the access connector resource instance rule cover classic compute?

No, not automatically.

The access connector's resource instance rule allows eligible access associated with its managed identity. It does not automatically allow network traffic originating from classic compute workers.

When Unity Catalog provides temporary SAS credentials, those credentials authorize storage operations, but they do not bypass the storage firewall.

managed identity configuration documentation separates the workspace network-access requirement from the managed identity's access requirement.

Also, same-region Azure Storage traffic cannot be reliably allowed using NAT Gateway public IP rules.

For your existing DBR 17.3 LTS classic job workloads, I would choose VNet injection with storage service endpoints or Private Link if you need to retain classic compute.

If serverless is compatible, NCC with private endpoints is another supported option that avoids modifying the managed VNet.

You can additionally confirm the failing request's authentication method and source using Azure Storage resource logs (StorageBlobLogs), if diagnostic logging is enabled.

There is no documented self-service method to bypass the managed VNet's deny assignment safely.