Performing an custom update to Synapse dedicated pool table from Databricks\PySpark

Syed1 Shabeer Ahmed 5 Reputation points
2023-11-13T09:52:09.7133333+00:00

Hi Team,

I have a requirement where i need to read a transformed file and update only few columns to a Synapse dedicated pool table. Following is the update stmt:

UPDATE   <SCHEMA.TABLE>
   SET   col_1 =
            CASE
               WHEN :active_flg = 'Y' THEN :exmpt_rsn_id
               ELSE null
            END,
         col_2 = 
             CASE
               WHEN :active_flg = 'Y' THEN 1
               ELSE 0
            END,
         AUDIT_DTTM = <curr_timestamp>}
 WHERE       col_3 = :time_day_id
         AND col_4 = :loc_id
         AND col_5 = '19'

Here :active_flg, :exmp_rsn_id, :time_day_id_and :loc_id are coming from a the transformed file which would be available in ADLS.

I was thinking of reading the file into dataframe, then add two new columns to the dataframe with the case when logic.

But, how can we update the Synapse table from the dataframe? Please help in solutionizing this.

Thanks,

Syed

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

Azure Databricks
Azure Databricks

An Apache Spark-based analytics platform optimized for Azure.


1 answer

Sort by: Oldest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    5 deleted comments

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.