Local Home requires POSIX when deploying Bitbucket on AKS using azurefile csi.
Asked by Thiago de Oliveira Santos:

I am deploying Bitbucket to AKS following the Atlassian installation guide: [Install Atlassian DC Helm Charts.](https://atlassian.github.io/data-center-helm-charts/userguide/INSTALLATION/)

For Local Home I am dynamically creating both pv and pvc using the default storage class azurefile-csi from [Dynamically create azure files pvs by using the built in storage classes](https://learn.microsoft.com/en-us/azure/aks/azure-files-csi#dynamically-create-azure-files-pvs-by-using-the-built-in-storage-classes).

For local home I am passing the follow values on *values.yaml*

```
localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: true

# -- Specify the name of the 'StorageClass' that should be used for the local-home
# volume claim.
#
storageClassName: "azurefile-csi"

# -- Specifies the standard K8s resource requests and/or limits for the local-home
# volume claims.
#
resources:
requests:
storage: 1Gi
```

I also tried to create a claim:

```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-local-home
namespace: bitbucket
spec:
accessModes:
- ReadWriteMany
storageClassName: "azurefile-csi" 
resources:
requests:
storage: 1Gi
```

and passed it as:  

```
localHome:

# Dynamic provisioning of local-home using the K8s Storage Classes
#
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic
# https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/
#
persistentVolumeClaim:

# -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically
# created for each pod based on the 'StorageClassName' supplied below.
#
create: false

...

# -- Static provisioning of local-home using K8s PVs and PVCs
#
# NOTE: Due to the ephemeral nature of pods this approach to provisioning volumes for
# pods is not recommended. Dynamic provisioning described above is the prescribed
# approach.
#
# When 'persistentVolumeClaim.create' is 'false', then this value can be used to define
# a standard K8s volume that will be used for the local-home volume(s). If not defined,
# then an 'emptyDir' volume is utilised. Having provisioned a 'PersistentVolume', specify
# the bound 'persistentVolumeClaim.claimName' for the 'customVolume' object.
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#static
#
customVolume:
 persistentVolumeClaim:
 claimName: "pvc-local-home"
```

Both resulted in the same error.

After deploying the helm chart, the pod with Bitbucket starts but never gets to a ready state due to the following error:

```
2024-01-08 17:52:21,922 ERROR [spring-startup]  c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started

org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshHealthcheck' defined in URL [jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/bitbucket-git-mesh-8.9.8.jar!/com/atlassian/stash/internal/scm/git/mesh/MeshHealthcheck.class]:

 Unsatisfied dependency expressed through constructor parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshClient' parameter 2; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; 

 nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 

 Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: 

 Unsatisfied dependency expressed through method 'meshRouter' parameter 5; 

 nested exception is org.springframework.beans.factory.BeanCreationException:

 Error creating bean with name 'sidecarManager': 

 Invocation of init method failed; 

 nested exception is java.nio.file.FileSystemException: 

 /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem:

 Operation not permitted

 at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)

 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:929)

 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)

 at javax.servlet.GenericServlet.init(GenericServlet.java:143)

 at java.base/java.lang.Thread.run(Thread.java:829)

 ... 18 frames trimmed

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshClient' parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 ... 6 common frames omitted

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 ... 6 common frames omitted

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 ... 6 common frames omitted

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)

 ... 5 common frames omitted

Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted

 at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

 at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)

 at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)

 at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 ... 5 common frames omitted
```

When checking the stack trace I realized that the error:  

```
Caused by: java.nio.file.FileSystemException: /var/atlassian/application-data/bitbucket/mesh/config/control-plane.pem: Operation not permitted
```

is being thrown when executing:  

```
at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)
```

```
at com.atlassian.bitbucket.mesh.util.KeyUtils.write(KeyUtils.java:382)

 at com.atlassian.bitbucket.mesh.util.KeyUtils.writePublicKey(KeyUtils.java:260)

 at com.atlassian.stash.internal.scm.git.mesh.MeshLauncher.start(MeshLauncher.java:121)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:272)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.internalStart(DefaultSidecarManager.java:223)

 at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:178)
```

It seems to be an issue when trying to set **POSIX**.  
[Azure files CSI](https://learn.microsoft.com/en-us/azure/aks/azure-files-csi) is not compliant with **POSIX** , for that an [NFS File Share](https://learn.microsoft.com/en-us/azure/aks/azure-files-csi#nfs-file-shares) has to be used.  
When running the helm install using **emptyDir** for *local home* the issue does not happen.  
I couldn't find any mention of **POSIX** in the section [5 Configure Persistent Storage](https://atlassian.github.io/data-center-helm-charts/userguide/INSTALLATION/#5-configure-persistent-storage) of the installation guide.  
Am I doing something wrong or a volume compliant with **POSIX** is needed in order to run Bitbucket properly on Kubernetes?


Answers:
author: Nic Brough -Adaptavist-
body: This is broadly what it says it is - you need to move to using a standard supportable file system

NFS can work with older Windows file systems like NTFS and FAT, and all Unix file systems like ext1-4, btr and so-on, because these are all Unix compatible, and can be made to behave as though they are POSIX compliant.


Tags: azure

bitbucket-server

data-center

install

server