vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Why can't I add non-root user access to persistent storage? 

Classic infrastructure

 What’s happening 

After you [add non-root user access to persistent storage](https://cloud.ibm.com/docs/containers?topic=containers-nonroot) or deploy a Helm chart with a non-root user ID specified, the user can't write to the mounted storage.

 Why it’s happening 

Your app deployment or Helm chart configuration specifies the [security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for the pod's fsGroup (group ID) and runAsUser (user ID). Generally, a pod's default security context sets [runAsNonRoot](https://kubernetes.io/docs/concepts/security/pod-security-policy/) so that the pod can't run as the root user. Because the fsGroup setting is not designed for shared storage such as NFS file storage, the fsGroup setting is not supported, and the runAsUser setting is automatically set to 2020. These default settings don't allow other non-root users to write to the mounted storage.

 How to fix it 

To allow a non-root user read and write access to a file storage device, you must allocate a [supplemental group ID](https://kubernetes.io/docs/concepts/security/pod-security-policy/) in a storage class, refer to this storage class in the PVC, and set the pod's security context with a runAsUser value that is automatically added to the supplemental group ID. When you grant the supplemental group ID read and write access to the file storage, any non-root user that belongs to the group ID, including your pod, is granted access to the file storage.

You can use one of the provided [gid storage classes](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_storageclass_reference) or create a custom storage class to define your own supplemental group ID.
For more information, see [Specifying how strings are spelled out](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-synthesis-paramsparams-spell-out-mode).

Known limitation with using the Ogg audio format with the Safari browser
: By default, the service returns audio in the Ogg audio format with the Opus codec (audio/ogg;codecs=opus). However, the Ogg audio format is not supported with the Safari browser. If you are using the the Text to Speech service with the Safari browser, you must specify a different format in which you want the service to return the audio.



* For more information about the available formats, see [Supported audio formats](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formatsformats-supported).
* For more information about specifying a format, see [Specifying an audio format](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formatsformats-specify).



Troubleshooting upgrade from version 4.0.x to version 4.5.x
: When you upgrade the Speech services from version 4.0.x to version 4.5.x, you might encounter an issue where the PostgreSQL pods become stuck in the Terminating state. If this problem occurs during your upgrade, perform the following steps to resolve the problem. The information and steps are also documented in Upgrading Watson Speech services from Version 4.0 to Version 4.5 in the Upgrading topic of [Watson Speech services on IBM Cloud Pak for Data](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=services-watson-speech).



1. Use the following command to identify pods that remain in the Terminating state:



oc get pods -n ${PROJECT_CPD_INSTANCE} -o wide | awk {'print $1'}



1. Use the following command to set the environment variable pods to include the list of pods that remain in the Terminating state:
Important: Delete only one pod at a time, and verify that the pod is rescheduled onto an edge node before you delete other pods.



1. Delete a pod. Example for if one of the webserver-lb NLB pods did not schedule to an edge node:

oc delete pod ibm-cloud-provider-ip-169-46-17-2-76fcb4965d-wz6dg
2. Verify that the pod is rescheduled onto an edge worker node. Rescheduling is automatic, but might take a few minutes. Example for the webserver-lb NLB that has an external IP address of 169.46.17.2:

oc describe nodes -l dedicated=edge | grep "169-46-17-2"

Example output

ibm-system ibm-cloud-provider-ip-169-46-17-2-76fcb4965d-wz6dg 5m (0%) 0 (0%) 10Mi (0%) 0 (0%)
ibm-system ibm-cloud-provider-ip-169-46-17-2-76fcb4965d-2z64r 5m (0%) 0 (0%) 10Mi (0%) 0 (0%)



7. Retrieve all existing ALBs in the cluster. Review the command output. For each ALB that has a Status of enabled, note the ALB ID and Build.

ibmcloud oc ingress alb ls --cluster <cluster_name_or_ID>

Example output

ALB ID Enabled State Type Load Balancer Hostname Zone Build Status
private-crc81nk5l10gfhdql4i3qg-alb1 true enabled private e9dd35e6-us-south.lb.test.appdomain.cloud us-south-3 ingress:1.1.1_1949_iks/ingress-auth: enabled
This value can't be set to the value of the related location's pod-subnet or service-subnet.

--service-subnet SUBNET
: Optional. Specify a custom subnet CIDR to provide private IP addresses for services. This option can be used only if you also enable Red Hat CoreOS with the --coreos-enabled option. All services that are deployed to the cluster are assigned a private IP address in the 172.20.0.0/16 range by default. You can avoid subnet conflicts with the network that you use to connect to your location by specifying a custom subnet CIDR that provides the private IP addresses for your services.
: The subnet must be specified in CIDR format with a size of at least /24, which allows a maximum of 255 services in the cluster, or larger. The subnet that you choose must be within one of the following ranges.



* 172.16.0.0 - 172.17.255.255
* 172.20.0.0 - 172.31.255.255
* 192.168.0.0 - 192.168.254.255
* 198.18.0.0 - 198.19.255.255



: Note that the pod and service subnets can't overlap. The pod subnet is in the 172.16.0.0/16 range by default. This value can't be set to the value of the related location's pod-subnet or service-subnet.

--pod-network-interface-selection METHOD
: Optional. The method for selecting the node network interface for the internal pod network. The available methods are can-reach and interface. This option can be used only if you also enable Red Hat CoreOS with the --operating-system option.



* To provide a direct URL or IP address, specify can-reach=<url> or can-reach=<ip_address>. If the network interface can reach the provided URL or IP address, this option is used. For example, use can-reach=www.exampleurl.com for specifying a URL and can-reach=172.19.0.0 for specifying an IP address.
In the output, look for blocks that have 0 IPS IN USE, which indicates that the block is not used by its assigned worker node.

calicoctl ipam show --show-blocks

In this example output, the 172.24.10.64/26 block has no IP addresses in use.

...
Block 172.24.10.64/26 64 0 (0%) 64 (100%) 
...
4. For each of these blocks, complete the following steps to release the block.



1. Verify that no pods currently use an IP address from the block.

oc get pods -A
2. Get the BlockAffinity for the block. Replace the periods and forward slash in the block with hyphens (-). For example, for the block 172.24.10.64/26, the format for the following command is 172-24-10-64-26.

oc get blockaffinity | grep <block>
3. Delete the BlockAffinity.

oc delete blockaffinity <block_affinity>
4. Get the IPAMBlock for the block. Replace the periods and forward slash in the block with hyphens (-). For example, for the block 172.24.10.64/26, the format for the following command is 172-24-10-64-26.

oc get ipamblock | grep <block>
5. Delete the IPAMBlock.

oc delete ipamblock <ipam_block>
6. If you did not lock the data store in step 2: Check that no pods were created directly before you deleted the BlockAffinity and IPAMBlock. If pods were created, you must delete the BlockAffinity and IPAMBlock for this block again. Then, delete any pods that use an IP address in this block by running oc delete pod <pod> so the pod is recreated with an IP address from a different block.

oc get pods -A
7. Repeat these steps for any other blocks that that have 0 IPS IN USE.
Simple If you don't need TLS encryption, create a simple route to handle non-encrypted HTTP traffic. 
 Passthrough When you want TLS connections to pass uninterruptedly from the client to your app pod, create a passthrough route. The router is not involved in TLS termination for encrypted HTTPS traffic, so the app pod must terminate the TLS connection. This type can also be used for HTTP/2 and for non-HTTP TLS endpoints. 
 Edge When your app pod is exposed on a non-encrypted HTTP endpoint, but you must handle encrypted HTTPS traffic, create an edge route. The TLS connection between the client and the router service is terminated, and the connection between the router service and your app pod is unencrypted. For more information, see the [Red Hat OpenShift edge route documentation](https://docs.openshift.com/container-platform/4.11/networking/routes/secured-routes.htmlnw-ingress-creating-an-edge-route-with-a-custom-certificate_secured-routes). 
 Re-encrypt When your app pod is exposed on an encrypted HTTPS endpoint and you must handle HTTPS traffic, create a re-encrypt route. The TLS connection between the client and the router service is terminated, and a new TLS connection between the router service and your app pod is created. For more information, see the [Red Hat OpenShift re-encrypt route documentation](https://docs.openshift.com/container-platform/4.11/networking/routes/secured-routes.htmlnw-ingress-creating-a-reencrypt-route-with-a-custom-certificate_secured-routes). 



If you don't need to use a custom domain, you can use an IBM-provided route hostname in the format <service_name>-<project>.<cluster_name>-<random_hash>-0000.<region>.containers.appdomain.cloud.





 Ingress controller health checks
You must add node affinity and tolerations to app pods. For more information about basic load balancing with version 1.0 NLBs, see [Components and architecture of an NLB 1.0](https://cloud.ibm.com/docs/openshift?topic=openshift-loadbalancer-aboutv1_planning).

As opposed to version 1.0 NLBs, version 2.0 NLBs don't use NAT when forwarding requests to app pods on other workers. When an NLB 2.0 routes a client request, it uses IP over IP (IPIP) to encapsulate the original request packet into another packet. This encapsulating IPIP packet has a source IP of the worker node where the load balancer pod is, which allows the original request packet to preserve the client IP as its source IP address. The worker node then uses direct server return (DSR) to send the app response packet to the client IP. The response packet skips the NLB and is sent directly to the client, decreasing the amount of traffic that the NLB must handle. For more information about DSR load balancing with version 2.0 NLBs, see [Components and architecture of an NLB 2.0](https://cloud.ibm.com/docs/openshift?topic=openshift-loadbalancer-aboutplanning_ipvs).





 Components and architecture of an NLB 1.0 

The TCP/UDP network load balancer (NLB) 1.0 uses Iptables, a Linux kernel feature to load balance requests across an app's pods.

Mixed protocol load balancer services are not supported in IBM Cloud Kubernetes Service. You cannot specify both TCP and UDP ports your in load balancer definition.



 Traffic flow in a single-zone cluster 

The following diagram shows how an NLB 1.0 directs communication from the internet to an app in a single-zone cluster.

Zoom

![Expose an app in a single-zone cluster by using an NLB 1.0.]
The VPN state is still ESTABLISHED and pods in the specified namespace can connect over the VPN.
* The Calico global network policies in the following steps don't prevent Kubernetes pods that use host networking from sending and receiving data over the VPN. When a pod is configured with host networking, the app running in the pod can listen on the network interfaces of the worker node that it is on. These host networking pods can exist in any namespace. To determine which pods have host networking, run kubectl get pods --all-namespaces -o wide and look for any pods that don't have a 172.30.0.0/16 pod IP address. If you want to prevent host networking pods from sending and receiving data over the VPN, you can set the following options in your values.yaml deployment file: local.subnet: 172.30.0.0/16 and enablePodSNAT: false. These configuration settings expose all the Kubernetes pods over the VPN connection to the remote network. However, only the pods that are located in the specified secure namespace are reachable over the VPN.



Before you begin



* [Deploy the strongSwan Helm chart](https://cloud.ibm.com/docs/containers?topic=containers-vpnvpn_configure) and [ensure that VPN connectivity is working correctly](https://cloud.ibm.com/docs/containers?topic=containers-vpnvpn_test).
* [Install and configure the Calico CLI](https://cloud.ibm.com/docs/containers?topic=containers-network_policiescli_install).



To limit VPN traffic to a certain namespace,



1. Create a Calico global network policy named allow-non-vpn-outbound.yaml. This policy allows all namespaces to continue to send outbound traffic to all destinations, except to the remote subnet that the strongSwan VPN accesses. Replace <remote.subnet> with the remote.subnet that you specified in the Helm values.yaml configuration file.
You can use the default configuration to start.

apiVersion: pod-security.admission.config.k8s.io/v1
kind: PodSecurityConfiguration
defaults:
enforce: "privileged"
enforce-version: "latest"
audit: "privileged"
audit-version: "latest"
warn: "privileged"
warn-version: "latest"
exemptions:
usernames: []
runtimeClasses: []
namespaces: []
2. Make the customizations that you want to use. Review the following the information about each section of the configuration. defaults : The defaults section of the configuration defines cluster-wide defaults for namespaces that do not have pod security labels. The fields can have the same values as the corresponding namespace labels.

exemptions
: The exemptions section of the configuration allows you to create pods that are otherwise prohibited because of the policy associated with a specific namespace. Exemption dimensions include the following cases.



* Usernames: Requests from users with an exempt authenticated (or impersonated) username are ignored. Usernames are typically IAM users or service accounts. For example usernames: [ "IAMuser@example.com" ]. You can also exempt the username associated with an admin kubeconfig client certificate. If you want to specify an admin kubeconfig certificate, the username is the CN component of the current client-certificate.

kubectl config view --minify -o jsonpath='{.users[0].user.client-certificate}'
/home/user/.bluemix/plugins/container-service/clusters/mycluster-cheku1620qbs90gq6mdg-admin/admin.pem

In the following example, the username is IBMid-27000xxxxx-admin-2023-05-12. If you get a new admin kubeconfig, the username might be different.
Restart each pod by deleting it. In the output of the previous step, the pod name and namespace are listed in each entry as data plane version: version.ProxyInfo{ID:"<pod_name>.<namespace>", IstioVersion:"1.18.0"}.

kubectl delete pod <pod_name> -n <namespace>





Want to change a ConfigMap setting?
: If you want to change a setting that you added to the ConfigMap, you can use a patch script. For example, if you added the istio-global-proxy-accessLogFile: "/dev/stdout" setting and later want to change it back to "", you can run kubectl patch cm managed-istio-custom -n ibm-operators --type='json' -p='[{"op": "add", "path": "/data/istio-global-proxy-accessLogFile", "value":""}]'.

Need to debug your customization setup?
: Check the logs for the addon-istio-operator (Istio version 1.10 or later) or managed-istio-operator (Istio version 1.9 or earlier) pod by running kubectl logs -n ibm-operators -l name=managed-istio-operator. The Istio operator validates and reconciles any custom Istio changes that you make.

If you disable the Istio add-on, the managed-istio-custom ConfigMap is not removed during uninstallation. When you re-enable the Istio add-on, your customized ConfigMap is applied during installation. If you don't want to re-use your custom settings in a later installation of Istio, you must delete the ConfigMap after you disable the Istio add-on by running kubectl delete cm -n ibm-operators managed-istio-custom. When you re-enable the Istio add-on, the default ConfigMap is applied during installation.





 Updating the Istio add-on
: Keep in mind that privileged containers can cause huge damage to the cluster and the underlying compute host if they become compromised. Try to limit the number of containers that run in privileged mode and consider changing the configuration for your app so that the app can run without advanced permissions.

If you want to block privileged containers from running in your cluster, consider setting up custom [pod security policies](https://cloud.ibm.com/docs/containers?topic=containers-pspcustomize_psp).

Apply OS security settings to pods
: You can add the [securityContext section](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to your pods to control the user ID and group ID that can run inside the container, or the user ID and group ID that owns the volume mount path. Setting a specific user ID helps facilitate a least privilege model. If the security context does not specify a user, Kubernetes automatically uses the user that is specified in the container image.
: If you want to use securityContext to set the runAsUser user ID or fsGroup group ID, consider using block storage when you [create persistent storage](https://cloud.ibm.com/docs/containers?topic=containers-block_storageadd_block). NFS storage does not support fsGroup, and runAsUsermust be set at the container level, not the pod level.

Set CPU and memory limits for containers
: Every container requires a specific amount of CPU and memory to properly start and to continue to run. You can define [Kubernetes resource requests and resource limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for your containers or pods to limit the amount of CPU and memory that they can consume. If no limits for CPU and memory are set, and the container is busy, the container uses all the resources that are available. This high consumption of resources might affect other containers on the worker node that don't have enough resources to properly start or run, and puts your worker node at risk for denial-of-service attacks.

Enforce policy-driven authentication
: Note that the pod and service subnets can't overlap. The pod subnet is in the 172.16.0.0/16 range by default. This value can't be set to the value of the related location's pod-subnet or service-subnet.

--pod-network-interface-selection METHOD
: Optional. The method for selecting the node network interface for the internal pod network. The available methods are can-reach and interface. This option can be used only if you also enable Red Hat CoreOS with the --operating-system option.



* To provide a direct URL or IP address, specify can-reach=<url> or can-reach=<ip_address>. If the network interface can reach the provided URL or IP address, this option is used. For example, use can-reach=www.exampleurl.com for specifying a URL and can-reach=172.19.0.0 for specifying an IP address.
* To choose an interface with a Regex string, specify interface=<regex_string>; for example, interface=eth.



--provider INFRASTRUCTURE_PROVIDER
: Optional. The name of the infrastructure provider to create the Satellite location in. Accepted values are aws, azure, gcp. If you include this option, you must also include the --provider-credential option.

--provider-region PROVIDER_REGION
: Optional. The name of the region in the infrastructure provider where you plan to create all the hosts for the Satellite location, such as us-east-1 in AWS. Consult your infrastructure provider for the region name. If you include this option, you must also include the --provider option.

--provider-credential PATH_TO_PROVIDER_CREDENTIAL
: Optional. The path to a JSON file on your local machine that has the credentials of the infrastructure provider for the Satellite location. The credential format is provider-specific. For more information, see [Providing Satellite with credentials to your infrastructure provider](https://cloud.ibm.com/docs/satellite?topic=satellite-infrastructure-plan).
Why can't my app access or write to PVCs? 

Classic infrastructure

When you mount a PVC to your pod, you might experience errors when accessing or writing to the PVC.



1.  List the pods in your cluster and review the status of the pod.

kubectl get pods
2.  Find the root cause for why your app can't access or write to the PVC.

kubectl describe pod <pod_name>

kubectl logs <pod_name>
3.  Review common errors that can occur when you mount a PVC to a pod.





File Storage for Classic error messages

 Symptom or error message                                                                                                                                                                Description                                                                                                                                                                                                                            Steps to resolve                                                                                                                                                             

 Your pod is stuck in a ContainerCreating state. MountVolume.SetUp failed for volume ... read-only file system.                                                                          The IBM Cloud infrastructure back end experienced network problems. To protect your data and to avoid data corruption, IBM Cloud automatically disconnected the file storage server to prevent write operations on NFS file shares.    See [File storage: File systems for worker nodes change to read-only](https://cloud.ibm.com/docs/containers?topic=containers-readonly_nodes)                                      
 write-permissiondon't have required permissioncan't create directory '/bitnami/mariadb/data': Permission denied                                                                         In your deployment, you specified a non-root user to own the NFS file storage mount path. By default, non-root users don't have write permission on the volume mount path for NFS-backed storage.                                      See [File storage: App fails when a non-root user owns the NFS file storage mount path](https://cloud.ibm.com/docs/containers?topic=containers-nonroot)  
 After you specified a non-root user to own the NFS file storage mount path or deployed a Helm chart with a non-root user ID specified, the user can't write to the mounted storage.     The deployment or Helm chart configuration specifies the security context for the pod's fsGroup (group ID) and runAsUser (user ID)                                                                                                     See [File storage: Adding non-root user access to persistent storage fails](https://cloud.ibm.com/docs/containers?topic=containers-cs_storage_nonroot)