Azure Data Factory Dataflows hung since backend update to Spark 3.5

Use the Data Flow activity’s detailed monitoring view to identify whether the run is stalled during cluster allocation, a transformation stage, or sink post-processing, then apply the corresponding low-change runtime adjustment before restructuring the flow.

1. Locate the stalled stage

Run the published pipeline with Trigger Now or Debug → Use Activity Runtime, rather than executing it through the shared Data Flow debug cluster. Microsoft recommends activity-level runs when resource pressure on a debug cluster may be involved.^1^

Then:

1. Open ADF Studio → Monitor → Pipeline runs.
2. Open the affected pipeline run and select the Data Flow activity.
3. Select the eyeglasses icon.
4. Record:
   • Cluster startup time.
   • The last stage showing progress.
   • Per-stage processing time and row counts.
   • Partition skewness.
   • Sink processing and post-processing times.
5. Open the activity output JSON and retain the activity run ID and partition timings.

The monitoring view distinguishes cluster startup, source reads, transformations, and sink writes. It also exposes skew and per-partition timing, which is important because one stalled or heavily skewed partition can make the overall run appear hung.^2^

2. Apply the smallest matching change

If cluster startup or allocation is stalled

Temporarily run the affected Data Flow on a custom Azure Integration Runtime with a larger cluster:

1. Go to Manage → Integration runtimes.
2. Edit or create the Azure IR used by the Data Flow activity.
3. Under Data flow runtime, increase the compute size by one level.
4. Assign that IR to the activity and run one controlled test.

More cores increase parallel processing capacity, although additional cores stop helping when the workload has fewer partitions than available cores.^3^

TTL is unlikely to resolve a run that remains active for 17 hours. It only reduces cold-start time for sequential Data Flow activities by retaining the cluster after a completed execution; it is unavailable with AutoResolve IR and is not recommended for predominantly parallel flows.^3^

If a join or aggregate stage is stalled

Check the monitoring view for partition skew. If confirmed:

• On the transformation immediately before the affected join or aggregate, open Optimize.
• Test Set partitioning → Round robin when there is no suitable distribution key, or Hash using a high-cardinality, well-distributed key.
• Avoid Single partition, which Microsoft describes as very slow and harmful to downstream transformations.^4^

If the workload is large and uses joins or aggregations, another low-code test is:

1. Open Manage → Integration runtimes.
2. Edit the custom IR.
3. Go to Data flow runtime → Compute Custom Properties.
4. Add Shuffle partitions and test an incremental increase from the default rather than making a large jump.

This setting is intended for known partition-memory scenarios and can itself cause failures when used unnecessarily. Increasing it will not help if the data remains skewed.^3^

If sink post-processing is stalled

Inspect whether the sink uses output to single file, file merging, move/delete operations, or pre/post commands. The monitoring view reports these under post-processing; single-file output can produce substantial merge time.^2^

As a minimal test, disable single-file output or file merging for one run and allow partitioned output.

If all stages progress but telemetry is expensive

Change the activity’s logging level from Verbose to Basic for a comparison run. Verbose mode collects telemetry at each partition and can be expensive; Basic retains transformation durations.^4^

3. Treat Spark 3.5 as unconfirmed until Microsoft correlates the runs

ADF Mapping Data Flows execute on ADF-managed Spark clusters, so the Spark runtime itself is not customer-managed.^5^ However, the public ADF release page currently identifies an update to Spark 3.4, not a published Spark 3.5 Mapping Data Flow rollout or known indefinite-hang issue.^6^

For the support case, include:

• Factory name and Azure region.
• Pipeline and activity run IDs for one last successful run and several hanging runs.
• Exact date/time when behavior changed.
• Integration Runtime name, compute type, and core count.
• Data Flow script or exported ARM/JSON definition.
• Screenshots or JSON showing the last active stage, partition skew, and row counts.
• Confirmation that the same definition completed under the previous observed runtime.
• Results of one larger-compute test and, where applicable, one partitioning or sink-output test.

Because repeated executions no longer complete, further blind retries are unlikely to add useful evidence. Microsoft’s troubleshooting guidance directs persistent backend-service behavior to customer support after normal execution testing fails.^1^

References

1. Troubleshoot mapping data flows - Azure Data Factory | Microsoft Learn
2. Monitor Data Flows
3. Optimizing performance of the Azure Integration Runtime
4. Mapping data flow performance and tuning guide - Azure Data Factory ...
5. Mapping data flows - Azure Data Factory | Microsoft Learn
6. What's new in Azure Data Factory