Securing args of the Python script in Azure Data Factory Custom Azure Batch Activity

11-4688 191 Reputation points
2024-07-29T07:58:14.4633333+00:00

Hi,

Imagine running a simple ADF Pipeline that consists of 2 steps:

User's image

  1. Fetching a sensitive value - like Storage Account URL - from Azure Key Vault
  2. Executing a Python script with argsparse where this value is set as an argument with the command
       @{activity('SAURL').output.value
       
    

Both Web Activity and Custom Activity have secure input and output values.

THE PROBLEM:

Even though it seems fine from ADF perdpective, when I move to Batch Account -> jobs -> tasks, select Task and move to 'Properties' section, I am able to see the full command line command with the value of my SAURL.

QUESTIONS:

  1. How bad is it from security perspective?
  2. What is the proper workaround? My current solution is to move the sensitive data to the code and call them with azure.keyvault. secrets library, but it forces me to hard-code the name of the Key-Vault which is also bad. Also it would require adjusting all scripts (from the argsparse versions to the ones calling AKV).
  3. Therefore - is there any option to hash the command line command in the task (from the Batch or ADF perspective)?
Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

0 comments No comments

1 answer

Sort by: Newest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    1 deleted comment

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.