Any Limitation on number of mounts of a storage container on different servers using blobfuse2 ?

Himanshu Singh 10 Reputation points
2024-10-04T22:53:13.2066667+00:00

I have 12 on-prem servers at different locations and they all run k3s on it. I have mounted the same Azure storage account container on all these servers using blobfuse2 (Documentation referred). Please note that all 12 servers use the same container SAS token to mount the same container.
All these servers act as individual clusters (makes it 12 clusters). Each cluster has 3 pods responsible for writing data on azure blob container. All three pods write data in their unique directory.

Problem:
The above setup was working fine on 10 servers/clusters which crashed after 2 weeks of live run with the below error. As of today, the above setup is running only on 5-7 servers at a time.

Logs from csi-blob-node pod and container named blob :

I1002 15:47:26.575930 13323 nodeserver.go:127] NodePublishVolume: volume mystorage_mycontainer1 mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount at /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount with mountOptions: [bind]

I1002 15:47:26.575943 13323 mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount)

E1002 15:47:26.577082 13323 mount_linux.go:230] Mount failed: exit status 32

Mounting command: mount

Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount

Output: mount: /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount does not exist. DRIVER INFORMATION:

Build Date: "2023-10-19T11:51:15Z"

Compiler: gc

Driver Name: blob.csi.azure.com

Driver Version: v1.23.1

Git Commit: d6e08f4b0bcddbe42db9169c1bf9461d25b91093

Go Version: go1.21.1

Platform: linux/amd64

Is there a limit on number of read-write mounts for a single container using blobfuse2 ? Can you please help me in identifying the root cause of the problem?

Any help would be much appreciated.

Azure Storage
Azure Storage

Globally unique resources that provide access to data management services and serve as the parent namespace for the services.

Azure Blob Storage
Azure Blob Storage

An Azure service that stores unstructured data in the cloud as blobs.


1 answer

Sort by: Newest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    4 deleted comments

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.