How to make Confluence POD/ container runAsNonRoot in AKS deployed through HELM charts?
Asked by rughvi8784:

Kubernetes cluster is present in Azure (AKS). Deployed CONFLUENCE and BITBUCKET using HEM CHARTS.

We are security hardening the pods in our cluster.

Running the following command shows BITBUCKET and CONFLUENCE pods violate the PodSecurity enforcement.

```
kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=restricted
```

Result of the above is as follows:

```
Warning: bitbucket-0 (and 4 other pods): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
Warning: confluence-0 (and 1 other pod): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, runAsUser=0, seccompProfile
```

The following HELM config values break the POD completely. POD goes into error state and will not run at all.

```
confluence:  
  containerSecurityContext:
    capabilities:
      drop:
        - ALL
    runAsNonRoot: true
```

What are the HELM config values to enforce POD security hardening?

Any help is highly appreciated.

Thanks in advance.


Answers:
author: Yevhen
body: [@rughvi8784](/t5/user/viewprofilepage/user-id/5349682) can you share the exact errors from pod events when you add containerSecurityContext?

comments: author: rughvi8784
body: Hi [@Yevhen](/t5/user/viewprofilepage/user-id/3965218)

For the below settings

```
confluence:  
  containerSecurityContext:
    capabilities:
      drop:
        - ALL
    runAsNonRoot: true
```

The get pod and logs are as below.

```
kubectl get pod
```

![GetPod.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288317i2CEF8C2F7479AC11/image-size/large?v=v2&px=999 "GetPod.PNG")

```
kubectl logs confluence-0
```

![LogsPod.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288319i06D470A1D468667C/image-size/large?v=v2&px=999 "LogsPod.PNG")


author: rughvi8784
body: Hi [@Yevhen](/t5/user/viewprofilepage/user-id/3965218)

For the below settings (which does not include runAsNonRoot:true)

```
confluence:  
  containerSecurityContext:
    capabilities:
      drop:
        - ALL
```

Logs for confluence-0 pod are as below:

```
kubectl logs confluence-0
```

![logspod_before.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288320i47A7EA64FDBD60DE/image-size/large?v=v2&px=999 "logspod_before.PNG")

Is this of any help?

Thanks


author: Yevhen
body: try adding fsGroup: 2002 to containerSecurityContext to allow Confluence user to write to local home.


author: rughvi8784
body: Hi [@Yevhen](/t5/user/viewprofilepage/user-id/3965218)

I have tried the below:

![Try-1.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288431iA53AFB735A6E32A2/image-size/large?v=v2&px=999 "Try-1.PNG")

But when I fire the command

```
kubectl describe pod confluence-0
```

I get the result as

![Conf_Cont_Err_1.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288433i717EA25A8AE41FD2/image-size/large?v=v2&px=999 "Conf_Cont_Err_1.PNG")

Regrds


author: Yevhen
body: Make sure you add runAsUser: 2002 to securityContext

You will also want to set <https://github.com/atlassian/data-center-helm-charts/blob/main/src/main/charts/confluence/values.yaml#L834> to true if you are running as non root.


author: rughvi8784
body: Hi [@Yevhen](/t5/user/viewprofilepage/user-id/3965218)

After setting the following

![Try-2.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288461i1450FC84DC4DEA21/image-size/large?v=v2&px=999 "Try-2.PNG")

I get the below.

![conf_Cont_Err_2.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288458i3424F08ED02322CF/image-size/large?v=v2&px=999 "conf_Cont_Err_2.PNG")

OutOfSync confluence application further details.

![conf_Cont_Err_OutOfSync.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288459iCFE6716F538C981B/image-size/large?v=v2&px=999 "conf_Cont_Err_OutOfSync.PNG")

Even though the application is OutOfSync, I can still log into CONFLUENCE portal.  
Unfortunately even after these changes POD SECURITY is being violated.

![Conf_Rest_Err.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/288463iEAA0ACEEE59BE10D/image-size/large?v=v2&px=999 "Conf_Rest_Err.PNG")


author: Yevhen
body: [@rughvi8784](/t5/user/viewprofilepage/user-id/5349682) Just keep adding missing restrictions to securityContext. Btw, the admission controller may not like init containers as well. Do you have nfs fixer init container enabled in volumes.sharedHome? You can try setting securityContext for the entire pod (confluence.securityContext)  

You need to figure out the missing parts of securityContext now that you know how to run the container as nonRoot.


author: rughvi8784
body: Hi [@Yevhen](/t5/user/viewprofilepage/user-id/3965218) we have enabled NFS fixer in volumes.sharedHome.


author: Yevhen
body: This container runs as root, maybe it makes your admission controller unhappy.


author: rughvi8784
body: [@Yevhen](/t5/user/viewprofilepage/user-id/3965218) how do I runAsNonRoot for init containers?

I can see the following config value

Confluence.securityContext - this is set now. Still the pod security VIOLATION happening.

Confluence.containerSecurityContext - this breaks the pod.

Synchrony.securityContext - not set at the moment.

Synchrony.containerSecurityContext - not set at the moment.


author: Yevhen
body: Unfortunately, securityContext for init containers can't be set. However, nfs-permission fixer init container can be disabled (it fixes permissions for shared home volume, and it's not always necessary)


author: rughvi8784
body: Hi [@Yevhen](/t5/user/viewprofilepage/user-id/3965218)

Some good news.

Following settings have been used

![conf_final_settings.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/289371i13A1ABBBCC38AFA0/image-size/large?v=v2&px=999 "conf_final_settings.PNG")

The POD security violations are no more for confluence.

When I run the command

```
k label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=restricted
```

It will not complain about Confluence. That is what we need.

Confluence pods are running and confluence application is fine.

I can log into confluence portal and everything seems fine.

But when I see the logs of confluence POD, I spotted an error.

![conf_final_err.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/289373i3C1A76A6161E9E98/image-size/large?v=v2&px=999 "conf_final_err.PNG")

Should I worry about this? Will it cause any issue?

What does that error represent?

Thanks


author: Yevhen
body: This does not look like a fatal error, and I am pretty sure you can ignore it.


author: rughvi8784
body: [@Yevhen](/t5/user/viewprofilepage/user-id/3965218)

Thanks a lot for your answers. JIRA, CONFLUENCE and BITBUCKET are working as expected. (Happy path is fine). We'll do more testing on these.

---

I am keen to understand more about NFSPERMISSIONFIXER. What is the use of this and when do we need that. For now we have disabled that.   
Do you think disabling that will affect POD functioning in any scenarios? Like in EKS or on premise Kubernetes cluster etc.

Also can you please shed some light on the performance impacts for disabling NFSPERMISSIONFIXER.

Thanks


author: Yevhen
body: [@rughvi8784](/t5/user/viewprofilepage/user-id/5349682) this init container has been added to address permissions issue that you \`might\` have with nfs PVs. You can disable it, and if products can run without fatal errors (caused by inability to write to shared home), then you're good to go.


Tags: confluence-server

security

server