Repository: ceph/ceph-csi

Issue #486: Failed to create cephfs  PVC
Created: 2019-07-22T08:28:43Z
Describe the bug
Failed to create cephfs PVC
Environment details

Image/version of Ceph CSI driver: canary
helm chart version
Kubernetes cluster version: 1.15.0
Logs

E0722 08:26:57.932549       1 volume.go:100] failed to create subvolume group csi, for the vol csi-vol-77ecd37e-ac5a-11e9-8a38-22008e3300de(an error occurred while running (597) ceph [fs subvolumegroup create myfs csi --mode 777 --pool_layout myfs-data0 -m 10.106.133.206:6789 -c /etc/ceph/ceph.conf -n client.admin --key=***stripped***]: exit status 22: 2019-07-22 08:26:57.543 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.543 7f97e93ba700 -1 AuthRegistry(0x7f97e4063b98) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.580 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-07-22 08:26:57.614 7f97e93ba700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 909, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 182, in handle_command
    return handler(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 208, in _cmd_fs_subvolumegroup_create
    return self.vc.create_subvolume_group(vol_name, group_name, mode=mode, pool=pool_layout)
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 305, in create_subvolume_group
    sv.create_group(spec, pool=pool, mode=self.octal_str_to_decimal_int(mode))
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 178, in create_group
    self._mkdir_p(path, mode)
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 58, in _mkdir_p
    self.fs.mkdir(subpath, mode)
  File "cephfs.pyx", line 801, in cephfs.LibCephFS.mkdir (/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1-636-g64f035c/rpm/el7/BUILD/ceph-14.2.1-636-g64f035c/build/src/pybind/cephfs/pyrex/cephfs.c:8613)
Error: (13, "error in mkdir 'volumes': error code 13")


Steps to reproduce
deploy cephfs plugin and create PVC

Duplicate issue #562: [cephfs csi] cephfs csi plugin cannot create volume normally
Created: 2019-08-19T13:08:18Z
Describe the bug
cephfs csi plugin cannot create volume normally
Environment details

Image/version of Ceph CSI driver

[root@ln-node5 ~]# docker images |grep csi
quay.io/cephcsi/cephcsi                    v1.1.0              408da8f56511        5 weeks ago         933MB
quay.io/k8scsi/csi-attacher                v1.2.0              eef7a9550ede        7 weeks ago         46.2MB
quay.io/k8scsi/csi-provisioner             v1.3.0              2422aa77f05f        8 weeks ago         50.6MB
quay.io/k8scsi/csi-snapshotter             v1.1.0              2775054017ba        4 months ago        44.3MB
quay.io/k8scsi/csi-node-driver-registrar   v1.1.0              a93898755322        4 months ago        15.8MB
quay.io/k8scsi/csi-node-driver-registrar   v1.0.2              8fed7d1b1015        7 months ago        20.1MB


helm chart version
not use helm
Kubernetes cluster version

[root@ln-node5 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}


Logs

I0819 12:46:36.981441       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cephfs-pvc", UID:"964d596f-4c5a-4f7e-8ffb-d3d37d44a3cc", APIVersion:"v1", ResourceVersion:"273738", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = an error occurred while running (1891) ceph [fs subvolumegroup create cephfs csi --mode 777 --pool_layout cephfs_data -m 10.150.7.89:6789,10.150.7.165:6789 -c /etc/ceph/ceph.conf -n client.admin --key=***stripped***]: exit status 22: 2019-08-19 12:46:36.226 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.226 7f24a3565700 -1 AuthRegistry(0x7f249c063b98) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2019-08-19 12:46:36.275 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.276 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.276 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.312 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.312 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.313 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:46:36.313 7f24a3565700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 909, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 182, in handle_command
    return handler(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 208, in _cmd_fs_subvolumegroup_create
    return self.vc.create_subvolume_group(vol_name, group_name, mode=mode, pool=pool_layout)
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 305, in create_subvolume_group
    sv.create_group(spec, pool=pool, mode=self.octal_str_to_decimal_int(mode))
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 178, in create_group
    self._mkdir_p(path, mode)
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 58, in _mkdir_p
    self.fs.mkdir(subpath, mode)
  File "cephfs.pyx", line 801, in cephfs.LibCephFS.mkdir (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/build/src/pybind/cephfs/pyrex/cephfs.c:8613)
Error: (13, "error in mkdir 'volumes': error code 13")


Steps to reproduce
Steps to reproduce the behavior:
Follow the steps in https://github.com/ceph/ceph-csi/blob/master/README.md and using
the template files under examples/cephfs/

use the examples/cephfs/plugin-deploy.sh deploy the csi plugin
setup config map according my ceph cluster info

[root@ln-node4 examples]# kubectl get cm ceph-csi-config -o yaml
apiVersion: v1
data:
  config.json: |-
    [
      {
        "clusterID": "33125608-9d11-4db9-b04a-a36ac282c1c4",
        "monitors": [
            "10.150.7.89:6789",
            "10.150.7.165:6789"
        ]
      }
    ]
kind: ConfigMap
metadata:
  creationTimestamp: "2019-08-19T09:43:10Z"
  name: ceph-csi-config
  namespace: default
  resourceVersion: "272876"
  selfLink: /api/v1/namespaces/default/configmaps/ceph-csi-config
  uid: 85f808c8-9bbe-4abe-a223-caff85594f71

my ceph cluster info:
[root@ln-node4 examples]# ceph fsid
33125608-9d11-4db9-b04a-a36ac282c1c4
[root@ln-node4 examples]# ceph mon dump
dumped monmap epoch 1
epoch 1
fsid 33125608-9d11-4db9-b04a-a36ac282c1c4
last_changed 2019-08-19 10:51:35.600929
created 2019-08-19 10:51:35.600929
min_mon_release 14 (nautilus)
0: [v2:10.150.7.89:3300/0,v1:10.150.7.89:6789/0] mon.ln-node6
1: [v2:10.150.7.165:3300/0,v1:10.150.7.165:6789/0] mon.ln-node7


setup the secret

[root@ln-node4 examples]# cat cephfs/secret.yaml 
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: default
stringData:
  userID: admin
  userKey: AQDEDlpdDojTARAAhTLBlun19TTVkeY5RFmylQ==
  adminID: admin
  adminKey: AQDEDlpdDojTARAAhTLBlun19TTVkeY5RFmylQ==

my ceph cluster info:
[root@ln-node4 examples]# ceph auth get client.admin
exported keyring for client.admin
[client.admin]
        key = AQDEDlpdDojTARAAhTLBlun19TTVkeY5RFmylQ==
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"


setup the storageclass.yaml

[root@ln-node4 examples]# cat cephfs/storageclass.yaml 
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  # String representing a Ceph cluster to provision storage from.
  # Should be unique across all Ceph clusters in use for provisioning,
  # cannot be greater than 36 bytes in length, and should remain immutable for
  # the lifetime of the StorageClass in use.
  # Ensure to create an entry in the config map named ceph-csi-config, based on
  # csi-config-map-sample.yaml, to accompany the string chosen to
  # represent the Ceph cluster in clusterID below
  clusterID: 33125608-9d11-4db9-b04a-a36ac282c1c4

  # CephFS filesystem name into which the volume shall be created
  fsName: cephfs

  # Ceph pool into which the volume shall be created
  pool: cephfs_data

  # The secrets have to contain user and/or Ceph admin credentials.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default

  # (optional) The driver can use either ceph-fuse (fuse) or
  # ceph kernelclient (kernel).
  # If omitted, default volume mounter will be used - this is
  # determined by probing for ceph-fuse
  # or by setting the default mounter explicitly via
  # --volumemounter command-line argument.
  # mounter: kernel
reclaimPolicy: Delete
mountOptions:
  - debug


create pvc  with the following pvc yaml

[root@ln-node4 examples]# cat cephfs/pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc


Wait a moment and see error

[root@ln-node4 rbd]# kubectl get pvc
NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
cephfs-pvc   Pending                                                                        csi-cephfs-sc   5m1s
rbd-pvc      Bound     pvc-443a1bc1-3489-4565-981d-f5f20e13e0f6   1Gi        RWO            csi-rbd-sc      8m54s

The detail errors
[root@ln-node4 rbd]# kubectl describe pvc cephfs-pvc |less
Name:          cephfs-pvc
Namespace:     default
StorageClass:  csi-cephfs-sc
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type     Reason              Age    From                                                                                     Message
  ----     ------              ----   ----                                                                                     -------
  Warning  ProvisioningFailed  6m37s  cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-0_c210c52a-c265-11e9-b93d-9ede73a2ea0a  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = an error occurred while running (163) ceph [fs subvolumegroup create cephfs csi --mode 777 --pool_layout cephfs_data -m 10.150.7.89:6789,10.150.7.165:6789 -c /etc/ceph/ceph.conf -n client.admin --key=***stripped***]: exit status 22: 2019-08-19 12:44:09.616 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.616 7f84a1c36700 -1 AuthRegistry(0x7f849c063b98) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2019-08-19 12:44:09.680 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.682 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.682 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.718 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.718 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.719 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:09.719 7f84a1c36700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 909, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 182, in handle_command
    return handler(inbuf, cmd)
  File "/usr/share/ceph/mgr/volumes/module.py", line 208, in _cmd_fs_subvolumegroup_create
    return self.vc.create_subvolume_group(vol_name, group_name, mode=mode, pool=pool_layout)
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 305, in create_subvolume_group
    sv.create_group(spec, pool=pool, mode=self.octal_str_to_decimal_int(mode))
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 178, in create_group
    self._mkdir_p(path, mode)
  File "/usr/share/ceph/mgr/volumes/fs/subvolume.py", line 58, in _mkdir_p
    self.fs.mkdir(subpath, mode)
  File "cephfs.pyx", line 801, in cephfs.LibCephFS.mkdir (/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/build/src/pybind/cephfs/pyrex/cephfs.c:8613)
Error: (13, "error in mkdir 'volumes': error code 13")
  Warning  ProvisioningFailed  6m34s  cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-0_c210c52a-c265-11e9-b93d-9ede73a2ea0a  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = an error occurred while running (379) ceph [fs subvolumegroup create cephfs csi --mode 777 --pool_layout cephfs_data -m 10.150.7.89:6789,10.150.7.165:6789 -c /etc/ceph/ceph.conf -n client.admin --key=***stripped***]: exit status 22: 2019-08-19 12:44:12.628 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.628 7f31119e1700 -1 AuthRegistry(0x7f310c063b98) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2019-08-19 12:44:12.691 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.697 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.697 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.730 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.731 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.731 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-08-19 12:44:12.731 7f31119e1700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Error EINVAL: Traceback (most recent call last):

Actual results
cannot create pvc normally, the pvc status was always pending, like this
[root@ln-node4 cephfs]# kubectl get pvc
NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
cephfs-pvc   Pending                                                                        csi-cephfs-sc   11m
rbd-pvc      Bound     pvc-443a1bc1-3489-4565-981d-f5f20e13e0f6   1Gi        RWO            csi-rbd-sc      15m


Expected behavior
create pvc normally
Additional context
The RBD example is OK with the same kubernetes cluster enviroment.

Closing comment (2019-08-20T06:58:21Z): This seems to be a duplicate of #486
The next stable release of Ceph nautilus v14.2.3 will have the fix
.
See, #486 (comment)
@scott0000 ,  can you try doing this for now,  https://github.com/ceph/ceph-csi/blob/master/docs/deploy-cephfs.md#configuration ?