An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Performing an custom update to Synapse dedicated pool table from Databricks\PySpark
Hi Team,
I have a requirement where i need to read a transformed file and update only few columns to a Synapse dedicated pool table. Following is the update stmt:
UPDATE <SCHEMA.TABLE>
SET col_1 =
CASE
WHEN :active_flg = 'Y' THEN :exmpt_rsn_id
ELSE null
END,
col_2 =
CASE
WHEN :active_flg = 'Y' THEN 1
ELSE 0
END,
AUDIT_DTTM = <curr_timestamp>}
WHERE col_3 = :time_day_id
AND col_4 = :loc_id
AND col_5 = '19'
Here :active_flg, :exmp_rsn_id, :time_day_id_and :loc_id are coming from a the transformed file which would be available in ADLS.
I was thinking of reading the file into dataframe, then add two new columns to the dataframe with the case when logic.
But, how can we update the Synapse table from the dataframe? Please help in solutionizing this.
Thanks,
Syed
Azure Synapse Analytics
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1 answer
Sort by: Newest
-
Deleted
This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
5 deleted comments
Comments have been turned off. Learn more