How to Ingest a table of xml data from on-prem DB2 warehouse to ASDLGen2 using ADF

Question

How to Ingest a table of xml data from on-prem DB2 warehouse to ASDLGen2 using ADF

Nhan, Phuong 0

Hi,

I am using ADF to incrementally load a table containing 2 XML columns from on-prem DB2 warehouse to ADSLGen2 in parquet format. In Copy activity, I use DB2 connector at source and parquet dataset at sink; I also use Query with XMLSERIALIZE for these 2 columns to convert and compress the data; I also explicitly mapped these 2 columns to String. This setup works for a small amount of records; if the record counts are more than 2 thousands, ADF just hang for hours and stuck. What should I change in the ADF pipeline to make it work, please?

Thank you

Himaja Y 545 Reputation points Microsoft External Staff Moderator

2026-09-16T14:29:34.7866667+00:00
Hi @Nhan, Phuong ,

Thank you for reaching out to the Microsoft Q&A.

The issue is likely related to processing large XML data rather than the ADF mapping itself. Since the load works for small datasets but hangs when processing more than 2,000 records, consider the following:

Check the execution time of the XMLSERIALIZE query directly in DB2.

Enable source partitioning and parallel copy in ADF.

Increase DIUs and parallel copy settings.

Validate whether some XML records are significantly larger than others.

As a test, load the data to CSV first instead of Parquet to determine if the bottleneck is in Parquet conversion.

Please share:

XML column size (average/max)

Azure IR or Self-Hosted IR

Copy activity monitoring metrics

This will help determine whether the bottleneck is in DB2 query execution, network transfer, or Parquet generation.
Himaja Y 545 Reputation points Microsoft External Staff Moderator

2026-09-17T16:18:09.3666667+00:00

Hi @Nhan, Phuong ,

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

1 answer

Your answer

Himaja Y 545 Reputation points Microsoft External Staff Moderator

2026-09-16T14:29:34.7866667+00:00

Hi @Nhan, Phuong ,

Thank you for reaching out to the Microsoft Q&A.

The issue is likely related to processing large XML data rather than the ADF mapping itself. Since the load works for small datasets but hangs when processing more than 2,000 records, consider the following:

Check the execution time of the XMLSERIALIZE query directly in DB2.

Enable source partitioning and parallel copy in ADF.

Increase DIUs and parallel copy settings.

Validate whether some XML records are significantly larger than others.

As a test, load the data to CSV first instead of Parquet to determine if the bottleneck is in Parquet conversion.

Please share:

XML column size (average/max)

Azure IR or Self-Hosted IR

Copy activity monitoring metrics

This will help determine whether the bottleneck is in DB2 query execution, network transfer, or Parquet generation.
Himaja Y 545 Reputation points Microsoft External Staff Moderator

2026-09-17T16:18:09.3666667+00:00

Hi @Nhan, Phuong ,

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Answer 1

Thank you for your responses.

Originally, with parquet at target, we did not use XMLSERIALIZE and the transfer was all good but... we ended up with a few broken XML records (like missing a closing bracket...). Then we tried it with XMLSERIALIZE and parquet; it works for a small set with a few records and hangs with a large set.

XML column size (average/max): 32MB max (I do not have the average though)
Azure IR or Self-Hosted IR: we use SHIR
Copy activity monitoring metrics as below One thing. We have just tried again with XMLSERIALIZE and Json at target. The pipeline was hanging again.