Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
Any Limitation on number of mounts of a storage container on different servers using blobfuse2 ?
I have 12 on-prem servers at different locations and they all run k3s on it. I have mounted the same Azure storage account container on all these servers using blobfuse2 (Documentation referred). Please note that all 12 servers use the same container SAS token to mount the same container.
All these servers act as individual clusters (makes it 12 clusters). Each cluster has 3 pods responsible for writing data on azure blob container. All three pods write data in their unique directory.
Problem:
The above setup was working fine on 10 servers/clusters which crashed after 2 weeks of live run with the below error. As of today, the above setup is running only on 5-7 servers at a time.
Logs from csi-blob-node pod and container named blob :
I1002 15:47:26.575930 13323 nodeserver.go:127] NodePublishVolume: volume mystorage_mycontainer1 mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount at /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount with mountOptions: [bind]
I1002 15:47:26.575943 13323 mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount)
E1002 15:47:26.577082 13323 mount_linux.go:230] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount
Output: mount: /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount does not exist.
DRIVER INFORMATION:
Build Date: "2023-10-19T11:51:15Z"
Compiler: gc
Driver Name: blob.csi.azure.com
Driver Version: v1.23.1
Git Commit: d6e08f4b0bcddbe42db9169c1bf9461d25b91093
Go Version: go1.21.1
Platform: linux/amd64
Is there a limit on number of read-write mounts for a single container using blobfuse2 ? Can you please help me in identifying the root cause of the problem?
Any help would be much appreciated.
Azure Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1 answer
Sort by: Oldest
-
Deleted
This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
4 deleted comments
Comments have been turned off. Learn more