Upload .txt and .sh files to databricks Workspace folder.

@Shriram Sethi

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

Upload .txt and .sh files to databricks Workspace folder.

Solution:

"We can utilize the Databricks API to upload .txt, .sh, or other file formats to the Databricks Workspace. This can be achieved through an inline script in the Azure pipeline or by referencing an external script file.

# Variables
$DATABRICKS_URL = "$(hostName)" #referred from environment variables
$TOKEN =  "$(PAT)" #referred from environment variables
$SOURCE_PATH_SCRIPT = "source_path/file_name.sh"
$DEST_PATH_SCRIPT = "/Workspace/folder/file_name.sh"
$SOURCE_PATH_TXT = "source_path/file_name.txt"
$DEST_PATH_TXT = "/Workspace/folder/file_name.txt"
$FILE_TYPE = "AUTO"  # AUTO will detect the file type

# Function to upload a file to Databricks workspace
function Upload-FileToDatabricks {
    param (
        [string]$sourcePath,
        [string]$destPath
    )

# Encode the file content to base64
    $content = [Convert]::ToBase64String([System.IO.File]::ReadAllBytes($sourcePath))
    
	# Create the JSON payload
    $jsonPayload = @{
        path = $destPath
        content = $content
        format = $FILE_TYPE
        overwrite = $true
    } | ConvertTo-Json
    
	# Make the API request
    Invoke-RestMethod -Method Post -Uri "$DATABRICKS_URL/api/2.0/workspace/import" -Headers @{Authorization = "Bearer $TOKEN"; "Content-Type" = "application/json"} -Body $jsonPayload
}

# Upload the script file
Upload-FileToDatabricks -sourcePath $SOURCE_PATH_SCRIPT -destPath $DEST_PATH_SCRIPT

# Upload the txt file
Upload-FileToDatabricks -sourcePath $SOURCE_PATH_TXT -destPath $DEST_PATH_TXT

The script mentioned above uploads .sh and .txt files to the Databricks Workspace. After uploading, you can reference the script in Databricks Compute and Workflow Compute. To ensure the init script runs when the cluster starts, update the Workflow JSON file accordingly."

"init_scripts": [
            {
              "workspace": {
                "destination": "/folder/file_name.sh"
              }
            }
          ]

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.