Stackgres Backup to Azure Blob Container Failing Due to Authentication Issue

Question

Stackgres Backup to Azure Blob Container Failing Due to Authentication Issue

Naveen Singh Padiyar 25

Hi All,

I have deployed Stackgres on an Azure AKS Cluster and am trying to schedule a backup on Stackgres to place the backup files onto an Azure Blob Container.

I have generated the SAS key for the container and used it to create a Backup secret on Stackgres. However, the scheduled backups are failing due to an authentication issue.

Here are the details:

PS C:\temp\Stackgres\mustang> kubectl get sgbackup -n myspace

NAME CLUSTER MANAGED STATUS

mustang-cluster-2025-04-17-14-15-08 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-15-13 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-15-30 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-15-56 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-30-07 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-30-12 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-30-29 mustang-cluster true Failed

mustang-cluster-2025-04-17-14-31-00 mustang-cluster true Failed

PS C:\temp\Stackgres\mustang> kubectl describe sgbackup mustang-cluster-2025-04-17-14-31-00 -n myspace

Name: mustang-cluster-2025-04-17-14-31-00

Namespace: myspace

Labels: stackgres.io/scheduled-backup=true

stackgres.io/scheduled-backup-job-name=mustang-cluster-backup-29081670

Annotations: stackgres.io/operatorVersion: 1.16.0

stackgres.io/scheduled-backup: true

API Version: stackgres.io/v1

Kind: SGBackup

Metadata:

Creation Timestamp: 2025-04-17T14:31:01Z

Generation: 4

Resource Version: 35047

UID: e1856089-c3ec-476d-86d6-f6e9e4b8ba94

Spec:

Managed Lifecycle: true

Reconciliation Timeout: 300

Sg Cluster: mustang-cluster

Status:

Backup Path: sgbackups.stackgres.io/myspace/mustang-cluster/2025-04-17-14-14-16/17

Process:

Failure: Backup failed while performing backup-push:

INFO: 2025/04/17 14:31:02.019993 Backup will be pushed to storage: default

INFO: 2025/04/17 14:31:02.027039 Doing full backup.

INFO: 2025/04/17 14:31:02.032253 Calling pg_start_backup()

INFO: 2025/04/17 14:31:02.171086 Initializing the PG alive checker (interval=1m0s)...

INFO: 2025/04/17 14:31:02.171121 Starting a new tar bundle

INFO: 2025/04/17 14:31:02.171150 Walking ...

INFO: 2025/04/17 14:31:02.171423 Starting part 1 ...

INFO: 2025/04/17 14:31:02.337208 Packing ...

INFO: 2025/04/17 14:31:02.341070 Finished writing part 1.

ERROR: 2025/04/17 14:31:02.386415 upload blob "sgbackups.stackgres.io/myspace/mustang-cluster/2025-04-17-14-14-16/17/basebackups_005/base_000000010000000000000008/tar_partitions/part_001.tar.lz4": write error: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====

Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

RequestId:dc7df33e-e01e-004c-35a5-af9efe000000

Time:2025-04-17T14:31:02.3696895Z, Details:

AuthenticationErrorDetail: The MAC signature found in the HTTP request 'WuMnPGypvsHk1ylG4j1zrVxixqDC8gFcHYtlxq6KOYA=' is not the same as any computed signature. Server used following string to sign: 'PUT

4357457

application/octet-stream

x-ms-date:Thu, 17 Apr 2025 14:31:02 GMT

x-ms-version:2020-10-02

/mustangstorage/mustangbackups/sgbackups.stackgres.io/myspace/mustang-cluster/2025-04-17-14-14-16/17/basebackups_005/base_000000010000000000000008/tar_partitions/part_001.tar.lz4

blockid:/qmbai43RMl9Qw75+Mi86wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==

comp:block'.

ERROR: 2025/04/17 14:31:02.386430 upload: could not upload 'base_000000010000000000000008/tar_partitions/part_001.tar.lz4'

ERROR: 2025/04/17 14:31:02.386432 upload blob "sgbackups.stackgres.io/myspace/mustang-cluster/2025-04-17-14-14-16/17/basebackups_005/base_000000010000000000000008/tar_partitions/part_001.tar.lz4": write error: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====

Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

RequestId:dc7df33e-e01e-004c-35a5-af9efe000000

Time:2025-04-17T14:31:02.3696895Z, Details:

AuthenticationErrorDetail: The MAC signature found in the HTTP request 'WuMnPGypvsHk1ylG4j1zrVxixqDC8gFcHYtlxq6KOYA=' is not the same as any computed signature. Server used following string to sign: 'PUT

4357457

application/octet-stream

x-ms-date:Thu, 17 Apr 2025 14:31:02 GMT

x-ms-version:2020-10-02

/mustangstorage/mustangbackups/sgbackups.stackgres.io/myspace/mustang-cluster/2025-04-17-14-14-16/17/basebackups_005/base_000000010000000000000008/tar_partitions/part_001.tar.lz4

blockid:/qmbai43RMl9Qw75+Mi86wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==

comp:block'.

ERROR: 2025/04/17 14:31:02.386435 Unable to continue the backup process because of the loss of a part 1.

command terminated with exit code 1

Job Pod: mustang-cluster-backup-29081670-g2g6n

Status: Failed

Sg Backup Config:

Base Backups:

Compression: lz4

Storage:

Azure Blob:

Azure Credentials:

Secret Key Selectors:

Access Key:

Key: AZURE_SAS_TOKEN

Name: mustang-backup-secret

Storage Account:

Key: AZURE_STORAGE_ACCOUNT

Name: mustang-backup-secret

Bucket: mustangbackups

Type: azureBlob

Events:

I tested the Storage account using azcopy to copy a file from my local machine, and it worked as expected.

I am not sure where the problem is and need some advice on how to troubleshoot this issue.

Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-17T18:22:26.73+00:00

Hi NaveenSinghPadiyar-4931 ,

I understand the issue with your Azure Blob Storage when trying to perform backups using Stackgres. I Suggest few steps you resolve this,

Ensure that the SAS token is correctly generated and has the necessary permissions. The token should include permissions for write, create, and add operations,

Make sure the storage account name and SAS token are correctly referenced in your Stackgres backup secret. Double-check the secret configuration for any typos or incorrect values

Ensure that the managed identity or service principal used by your AKS cluster has the Storage Blob Data Contributor role assigned to the storage account.

If your storage account has firewall rules enabled, ensure that the IP range of your AKS cluster is whitelisted

Sometimes, updating the Azure Storage SDK or tools like azcopy can resolve authentication issues, verify that the SAS token has not expired. If it has, generate a new token with a longer duration

The error message indicates a mismatch in the MAC signature. Ensure that the request headers and signature are correctly formed. You might need to review the way headers are being set in your backup configuration

For more information:

https://github.com/Azure/azure-cli/issues/15232

https://stackoverflow.com/questions/79525214/azure-blob-storage-this-request-is-not-authorized-to-perform-this-operation

Please do consider to “up-vote” wherever the information provided helps you, this can be beneficial to other community members. and please let us know if you have any further queries. I’m happy to assist you further.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-18T17:47:17.2433333+00:00

Hi NaveenSinghPadiyar-4931 ,

Your AKS cluster has proper DNS configuration. The error lookup dcQzkPzhtRUwzCVXYDtmlDfR8MO7rSQMbNvtX on 10.0.0.10:53: no such host indicates that the DNS server is unable to resolve the hostname. Verify that your cluster's DNS settings are correctly configured and Check if there are any network policies or firewall rules that might be blocking DNS resolution or access to Azure Blob Storage.

Check the storage account name in your secret configuration. Ensure that it matches the actual storage account name in Azure and Verify that the secret keys (accessKeyId and secretAccessKey) are correctly encoded in base64 and properly referenced in your SGObjectStorage configuration.

Ensure that the URL format for Azure Blob Storage is correct. It should follow the pattern https://<storage_account>.blob.core.windows.net/<container>/<blob>

Confirm that the storage account has the necessary permissions for the operations you are trying to perform.
Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-21T18:03:34.0066667+00:00

Hi NaveenSinghPadiyar-4931 ,

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be glad to assist you.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-21T22:25:41.39+00:00

Hi NaveenSinghPadiyar-4931 ,

Glad to know that your issue has resolved, using the correct access key resolved the issue. It's always satisfying to see things working as expected after troubleshooting.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

1 answer

Your answer

Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-17T18:22:26.73+00:00

Hi NaveenSinghPadiyar-4931 ,

I understand the issue with your Azure Blob Storage when trying to perform backups using Stackgres. I Suggest few steps you resolve this,

Ensure that the SAS token is correctly generated and has the necessary permissions. The token should include permissions for write, create, and add operations,

Make sure the storage account name and SAS token are correctly referenced in your Stackgres backup secret. Double-check the secret configuration for any typos or incorrect values

Ensure that the managed identity or service principal used by your AKS cluster has the Storage Blob Data Contributor role assigned to the storage account.

If your storage account has firewall rules enabled, ensure that the IP range of your AKS cluster is whitelisted

Sometimes, updating the Azure Storage SDK or tools like azcopy can resolve authentication issues, verify that the SAS token has not expired. If it has, generate a new token with a longer duration

The error message indicates a mismatch in the MAC signature. Ensure that the request headers and signature are correctly formed. You might need to review the way headers are being set in your backup configuration

For more information:

https://github.com/Azure/azure-cli/issues/15232

https://stackoverflow.com/questions/79525214/azure-blob-storage-this-request-is-not-authorized-to-perform-this-operation

Please do consider to “up-vote” wherever the information provided helps you, this can be beneficial to other community members. and please let us know if you have any further queries. I’m happy to assist you further.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-18T17:47:17.2433333+00:00

Hi NaveenSinghPadiyar-4931 ,

Your AKS cluster has proper DNS configuration. The error lookup dcQzkPzhtRUwzCVXYDtmlDfR8MO7rSQMbNvtX on 10.0.0.10:53: no such host indicates that the DNS server is unable to resolve the hostname. Verify that your cluster's DNS settings are correctly configured and Check if there are any network policies or firewall rules that might be blocking DNS resolution or access to Azure Blob Storage.

Check the storage account name in your secret configuration. Ensure that it matches the actual storage account name in Azure and Verify that the secret keys (accessKeyId and secretAccessKey) are correctly encoded in base64 and properly referenced in your SGObjectStorage configuration.

Ensure that the URL format for Azure Blob Storage is correct. It should follow the pattern https://<storage_account>.blob.core.windows.net/<container>/<blob>

Confirm that the storage account has the necessary permissions for the operations you are trying to perform.
Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-21T18:03:34.0066667+00:00

Hi NaveenSinghPadiyar-4931 ,

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be glad to assist you.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Keshavulu Dasari 4,930 Reputation points Microsoft External Staff Moderator

2025-04-21T22:25:41.39+00:00

Hi NaveenSinghPadiyar-4931 ,

Glad to know that your issue has resolved, using the correct access key resolved the issue. It's always satisfying to see things working as expected after troubleshooting.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 1

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more