Upload .txt and .sh files to databricks Workspace folder.

Question

Upload .txt and .sh files to databricks Workspace folder.

Shriram Sethi 5

I am trying to upload .txt and .sh files in the databricks workspace folder so that while starting a cluster the init script can be referred from the workspace folder, manually I am able to create the files but from azure devops pipeline its not uploading

Chandra Boorla 15,180 Reputation points Microsoft External Staff Moderator

2025-03-28T16:48:46.4533333+00:00

@Shriram Sethi

Just checking in to see if the below answer provided by @ Vinodh247 helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sunny Liu 0 Reputation points

2025-08-07T05:28:53.5566667+00:00

This was working well until today when I'm getting the error

{

"error_code": "RESOURCE_ALREADY_EXISTS",

"message": "Node named <some file path> already exists"

}

This is only happening for non-notebook files (e.g. json). overwrite = true is set and is not being honored.

Still works for notebook files (python, scala, r, ipynb)

Was there any updates in the background to API2.0? Can't see any change logs or documentation

3 answers

Your answer

Chandra Boorla 15,180 Reputation points Microsoft External Staff Moderator

2025-03-28T16:48:46.4533333+00:00

@Shriram Sethi

Just checking in to see if the below answer provided by @ Vinodh247 helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sunny Liu 0 Reputation points

2025-08-07T05:28:53.5566667+00:00

This was working well until today when I'm getting the error

{

"error_code": "RESOURCE_ALREADY_EXISTS",

"message": "Node named <some file path> already exists"

}

This is only happening for non-notebook files (e.g. json). overwrite = true is set and is not being honored.

Still works for notebook files (python, scala, r, ipynb)

Was there any updates in the background to API2.0? Can't see any change logs or documentation

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

To upload .txt and .sh files to a Databricks Workspace folder via Azure DevOps pipeline, you cannot use the DBFS CLI for Workspace uploads. Instead, use the Databricks REST API or the Databricks CLI (v0.205+) with workspace commands.

Preferred Approach: Use Databricks CLI with workspace API

Install Databricks CLI (v0.205 or later) in your pipeline:

Make sure your pipeline agent installs the latest CLI and sets up authentication. Use Databricks PAT (Personal Access Token) as DATABRICKS_TOKEN.

script: | pip install databricks-cli --upgrade databricks configure --token <<EOF $(DATABRICKS_HOST) $(DATABRICKS_TOKEN) EOF displayName: 'Install and Configure Databricks CLI'

2.Upload files to Workspace path

Example: Upload a shell script to /Workspace/Shared/init/init.sh. Make sure init.sh and myfile.txt are in your repository or available in the working directory.

script: | databricks workspace mkdirs /Shared/init databricks workspace import init.sh /Shared/init/init.sh --format AUTO --language SHELL databricks workspace import myfile.txt /Shared/init/myfile.txt --format AUTO --language TEXT displayName: 'Upload files to Databricks Workspace'

Workspace path is not the same as DBFS. Init scripts in Workspace can be referred using the path: /Workspace/Shared/init/init.sh
Use workspace path when cluster init script source is Workspace in UI or JSON.

Validate: Cluster JSON Snippet (Workspace-based init script). You can use this in your cluster definition (via REST API or Terraform) to attach the init script.

"init_scripts": [
  {
    "workspace": {
      "destination": "/Shared/init/init.sh"
    }
  }
]

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.

Chandra Boorla 15,180 Reputation points Microsoft External Staff Moderator

2025-04-01T17:15:57.32+00:00

@Shriram Sethi

Just checking in to see if the above answer provided by @ Vinodh247 helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Shriram Sethi 5 Reputation points

2025-04-02T09:45:32.28+00:00

Thank you @Vinodh247-1375 for the detailed explanation. The above solution might be working in the latest databricks CLI.
But when I tried the above databricks commands in the Azure Pipeline. Azure CLI and in local CLI where databricks CLI is installed. It didn't work and said the existing databricks CLI is deprecated.

I tried this below command

pip install databricks-cli --upgrade

still it didn't upgrade.

Instead of using the databricks cli I tried to use the databricks API, it is working perfectly.

I have mentioned the solution in the Answer section below.
Vinodh247-1375 44,281 Reputation points Volunteer Moderator

2025-04-02T16:39:17.2566667+00:00

Thanks for taking time to share the answer, this really helps!

Answer 2

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

1 deleted comment

Comments have been turned off. Learn more

Answer 3

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more