Writing binary data to ADLS in Synapse notebook

Hi Ljubo Jurkovic,

Thanks for reaching out to Microsoft Q&A.

Glad to know that you were able to figure this out and thanks for sharing the rootcause. By the way, Microsoft Q&A community has a policy that "The question author cannot accept their own answer, they can only accept answers by others.". Hence request you to accept this as an answer so we can close this thread. I am summarizing the issue and root cause you have found for it.

Issue Summary:

Write binary data to the ADLS in Synapse notebook using Pyspark? I retrieve data via an API call where the content type is application/excel (basically binary data) and want to save it to the ADLS, a specific location like this:

abfss://<container_name>@full_storage_account_name/<path_to_file>

Solution:

OP finally informed that he can't save the data as parquet because the original content format won't be preserved. He decided to use the Azure Function to do the API calls and save the content to the ADLS followed by the data next steps.

Workaround shared by op:

used C# libraries to issue the API call and save the content as a blob in the ADLS. var client = new HttpClient(); var accessToken = await GetToken(); var request = new HttpRequestMessage(HttpMethod.Get, url); request.Headers.Add("Authorization", "Bearer " + accessToken); var response = await client.SendAsync(request); response.EnsureSuccessStatusCode(); var blobContainerClient = new BlobContainerClient(connectionString, container); var responseContent = await response.Content.ReadAsByteArrayAsync(); Stream responseContentStream = new MemoryStream(responseContent); string filePath = Path.Combine(folder, surveyName + ".xlsx").Replace("", "/"); var blobClient = blobContainerClient.GetBlobClient(filePath); await blobClient.UploadAsync(responseContentStream, overwrite: true); published the Azure Function and call it from the Synapse pipeline using the Web activity.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Hi @Ljubo Jurkovic
Welcome to Microsoft Q&A platform and thanks for posting your question here.

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Issue: I need to check if particular SQL database table contains variable currentID.

Is it possible to write binary data to the ADLS in Synapse notebook using Pyspark? I retrieve data via an API call where the content type is application/excel (basically binary data) and want to save it to the ADLS, a specific location like this:

abfss://<container_name>@full_storage_account_name/<path_to_file>

It seems like a trivial task, but I can't find any examples of this.

Solution:
I can't save the data as parquet because the original content format won't be preserved. Anyway, I decided to use the Azure Function to do the API calls and save the content to the ADSL. After that I can do with the data next steps.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.