vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If the service does not have any backing resources, or if troubleshooting the pods does not resolve the issue, remove the mutating or validating webhook configuration identified earlier.

kubectl delete validatingwebhookconfiguration NAME

kubectl delete mutatingwebhookconfiguration NAME



6. Retry the cluster master operation, such as updating the cluster.
7. If you still see the error, you might have worker node or network connectivity issues.



* [Worker node troubleshooting](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodes).
* Make sure that the webhook can connect to the Kubernetes API server in the cluster master. For example, if you use Calico network policies, security groups, or some other type of firewall, set up your [classic](https://cloud.ibm.com/docs/containers?topic=containers-firewall) or [VPC](https://cloud.ibm.com/docs/containers?topic=containers-vpc-firewall) cluster with the appropriate access.
* If the webhook is managed by an add-on that you installed, uninstall the add-on. Common add-ons that cause webhook issues include the following:



* [Portieris](https://cloud.ibm.com/docs/openshift?topic=openshift-imagesportieris-image-sec)
* [Istio](https://cloud.ibm.com/docs/containers?topic=containers-istioistio_uninstall).





8. Re-create the webhook or reinstall the add-on.
Why do I get an error that my PodSecurityConfiguration is not valid? 

  What’s happening 

When you use the ibmcloud ks cluster master pod-security set command with the --config-file option, you see an error similar to the following example.

The 'configuration' field is not a valid Kubernetes PodSecurityConfiguration setting. See 'http://ibm.biz/iks-psa-config' for more information.

  Why it’s happening 

There is a problem with the pod security configuration file. The content of this file is described in [Pod Security Admission plug-in configuration](https://cloud.ibm.com/docs/containers?topic=containers-pod-security-admission).

Types of errors can include:



*  YAML formatting errors such as indentation or using tab characters instead of spaces for indentation.
*  The apiVersion or kind fields are missing or have incorrect values.
*  A field name is not spelled correctly.
*  A field has an incorrect value.



  How to fix it 

Check the configuration file for these kinds of errors. Correct any errors you find and try the command again.
Make sure to review the terms of the [license agreement](https://docs.docker.com/subscription/docker-desktop-license-agreement) before proceeding with the installation. Other options include [Podman](https://podman.io/get-started).
2. If you don't have one, sign up for a free account at [https://hub.docker.com/signup](https://hub.docker.com/signup).
3. Download and install Docker Desktop on Mac from [https://docs.docker.com/docker-for-mac/install/](https://docs.docker.com/docker-for-mac/install/).
4. Verify the installation with:

docker --version
docker run hello-world







 kubectl 



1. Download kubectl from

[https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-macos](https://kubernetes.io/docs/tasks/tools/install-kubectl/install-kubectl-on-macos).
2. Make the kubectl binary executable.

chmod +x ./kubectl
3. Move the binary to your PATH.

sudo mv ./kubectl /usr/local/bin/kubectl
4. Verify the installation with:

kubectl version --client=true







 oc 



1. Download the latest 4.x OpenShift CLI (oc) from [https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/).
2. Extract openshift-client-mac.tar.gz:
Why does mounting Block Storage for Classic fail with a file system check error? 

Classic infrastructure

  What’s happening 

When you view pod logs for pods that are using Block Storage for Classic, you see an error similar to the following example.

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.(i.e., without -a or -p
options)

  Why it’s happening 

The disk is corrupted.

  How to fix it 

Run the following commands to patch the PV that the pod is mounting.



1.  Get the details of the pod that is failing and make a note of the PV and PVC it is mounting.

kubectl get pods
2.  Patch the PV to add the "ibm.io/autofix-fsckErr":"true" annotation.

kubectl patch pv PVNAME -p '{"metadata": {"annotations":{"ibm.io/autofix-
fsckErr":"true"}}}'
3.  Delete and re-create your app pod to retry mounting the PV. After deleting and re-creating the pod, Kubernetes retries the mount operation and the Block Storage for Classic driver pod tries to auto fix the issue based on the "ibm.io/autofix-fsckErr":"true" annotation.
4.  After the pod is successfully deployed and running, run the following command to edit the PV.

kubectl edit pv PVNAME
5.  Remove the "ibm.io/autofix-fsckErr: true" annotation from the PV and save it.
6.  If the issue persists, contact support. Open a [support case](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar). In the case details, be sure to include any relevant log files, error messages, or command outputs.
(https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-pendingts-app-pod-pending)

[Why are pods in a CrashLoopBackOff status?](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-crashloopts-app-pod-crashloop)

[Why do pods repeatedly fail to restart or are unexpectedly removed?](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-failts-app-pod-fail)



* [Fixing container resource limits](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-failpod-fail-resource-limits)
* [Fixing pod replacement by higher priority pods](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-failpod-fail-higher-priority)



[Why does binding a service to a cluster results in service not found error?](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-svc-bind-not-foundts-app-svc-bind-not-found)



* [Cluster user](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-svc-bind-not-foundservice-bind-cluster-user)
* [As the account admin](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-svc-bind-not-foundservice-bind-account-admin)



[Why does binding a service to a cluster results in service does not support service keys error?](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-svc-keyts-app-svc-key)
Optional: Back up the webhooks, especially if you don't know how to reinstall the webhook.

oc get mutatingwebhookconfiguration <name> -o yaml > mutatingwebhook-backup.yaml

oc get validatingwebhookconfiguration <name> -o yaml > validatingwebhook-backup.yaml
5. Check the status of the related service and pods for the webhook.



1. Check the service Type, Selector, and Endpoint fields.

oc describe service -n <namespace> <service_name>
2. If the service type is ClusterIP, check that the Konnectivity pod is in a Running status so that the webhook can connect securely to the Kubernetes API in the cluster master. If the pod is not healthy, check the pod events, logs, worker node health, and other components to troubleshoot.



* Check the Konnectivity agent pods.

oc describe pods -n kube-system -l app=konnectivity-agent



3. If the service does not have an endpoint, check the health of the backing resources, such as a deployment or pod. If the resource is not healthy, check the pod events, logs, worker node health, and other components to troubleshoot. For more information, see [Debugging app deployments](https://cloud.ibm.com/docs/openshift?topic=openshift-debug_apps).

oc get all -n my-service-namespace -l <key=value>
4. If the service does not have any backing resources, or if troubleshooting the pods does not resolve the issue, remove the mutating or validating webhook configuration identified earlier.

oc delete validatingwebhookconfiguration NAME

oc delete mutatingwebhookconfiguration NAME



6. Retry the cluster master operation, such as updating the cluster.
7. If you still see the error, you might have worker node or network connectivity issues.
Known OpenShift issues For information on actions that might be required, review the [OpenShift version 4.12 known issues](https://docs.openshift.com/container-platform/4.12/release_notes/ocp-4-12-release-notes.htmlocp-4-12-known-issues). 
 Pod security admission synchronization If an Operator is installed in a user-created openshift-* namespace, synchronization is turned on by default after a cluster service version (CSV) is created in the namespace. The synchronized label inherits the permissions of the service accounts in the namespace. See [Controlling pod security admission synchronization](https://docs.openshift.com/container-platform/4.12/authentication/understanding-and-managing-pod-security-admission.htmlsecurity-context-constraints-psa-opting_understanding-and-managing-pod-security-admission) for actions that might be required. 
 Application updates required for natPortRange changes. If your app requires a larger number of egress network connections from pod-network pods to a destination that is external to the cluster, you might need to make updates. This applies if your app has more than 30,000 egress connections open on a single worker node simultaneously, or over 30,000 egress connections open on a single worker node within a few minutes. For more information, see [Why am I running out of SNAT ports for egress connections from pods in my cluster?](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-network-snat-125).
* [Fixing registry quota issues](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-container-startregitry-quota)
* [Fixing IP address issues](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-container-startcalico-ips)



* [Step 1: Releasing individual IP addresses](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-container-startindividual-ips)
* [Step 2: Releasing IP address blocks](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-container-startreleasing-individual-ips)





[Why do my pods fail to deploy after applying a pod security policy?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pspts-app-psp)

[Why do pods remain in pending state?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-pendingts-app-pod-pending)

[Why do pods repeatedly fail to restart or are unexpectedly removed?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-failts-app-pod-fail)



* [Fixing container resource limits](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-failpod-fail-resource-limits)
* [Fixing pod replacement by higher priority pods](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-failpod-fail-higher-priority)



[Why does binding a service to a cluster result in a same name error?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-svc-bind-namets-app-svc-bind-name)

[Why does binding a service to a cluster results in service not found error?]
Why are pods in a CrashLoopBackOff status? 

Virtual Private Cloud

Classic infrastructure

  What’s happening 

Your pods are in a CrashLoopBackOff status.

  Why it’s happening 

When you try to deploy an app that works on community Kubernetes platforms, you might see this status or a related error message because Red Hat OpenShift sets up stricter security settings by default than community Kubernetes.

  How to fix it 

Make sure that you review the [common scenarios where you might need to modify your apps](https://cloud.ibm.com/docs/openshift?topic=openshift-plan_deployopenshift_move_apps_scenarios) and follow the docs in the [Moving your apps to OpenShift topic](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy_appopenshift_move_apps).
* The underlying resource that issues the request is missing or unhealthy, such as a Kubernetes service, endpoint, or pod.
* The webhook is part of an add-on or other plug-in application that did not install correctly or is unhealthy.
* Your cluster might have a networking connectivity issue that prevents the webhook from communicating with the Kubernetes API server in the cluster master.



 How to fix it 

Identify and restore the resource that causes the broken webhook.



1. Create a test pod to get an error that identifies the broken webhook. The error message might have the name of the broken webhook. If the webhook test passes, then the failure may have been temporary and can be retried.

oc run webhook-test --image registry.ng.bluemix.net/armada-master/pause:3.2 -n ibm-system

In the following example, the webhook is trust.hooks.securityenforcement.admission.cloud.ibm.com.

Error from server (InternalError): Internal error occurred: failed calling webhook "trust.hooks.securityenforcementadmission.cloud.ibm.com": Post https://ibmcloud-image-enforcement.ibm-system.svc:443/mutating-pods?timeout=30s: dialtcp 172.21.xxx.xxx:443: connect: connection timed out
2. Get the name of the broken webhook.



* If the error message has a broken webhook, replace trust.hooks.securityenforcement.admission.cloud.ibm.com with the broken webhook that you previously identified.

oc get mutatingwebhookconfigurations,validatingwebhookconfigurations -o jsonpath='{.items[?(@.webhooks].name=="trust.hooks.securityenforcement.admission.cloud.ibm.com")].metadata.name}{"n"}'

Example output

image-admission-config
"subject": "docker://us.icr.io/cocoa-e2e/hello-compliance-app-test:20220401083914-master-76c13d7e9fe67379f04319c8e8095b1cc623acae@sha256:090ab641c5371e48cf263119162140e3c925b8041e0220f1397528da2828a30d",
"subjects": [
"docker://us.icr.io/cocoa-e2e/hello-compliance-app-test:20220401083914-master-76c13d7e9fe67379f04319c8e8095b1cc623acae@sha256:090ab641c5371e48cf263119162140e3c925b8041e0220f1397528da2828a30d"
],
"artifacts": [
{
"hash": "4e2cf12ffdf5438666be334106302f3e40cd0152ee1f26002cbc8bbcb5470796",
"url": "https://github.ibm.com/cocoa-test/e2e-compliance-evidence-locker-20220331113709516/blob/master/raw/attachments/4e/2cf12ffdf5438666be334106302f3e40cd0152ee1f26002cbc8bbcb5470796/content"
}
],
"issues": [
"https://github.ibm.com/cocoa-test/e2e-compliance-incident-issues-20220331113709516/issues/6",
"https://github.ibm.com/cocoa-test/e2e-compliance-incident-issues-20220331113709516/issues/7",
This results in a several second disruption to pod networking while the BGP password configuration is applied.



Changes to make before you update the master to Kubernetes 1.26

 Type Description 

 Unsupported: Beta version of the FlowSchema and PriorityLevelConfiguration API Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1beta2 API version, available since Kubernetes version 1.23. For more information, see [Deprecated API Migration Guide - v1.26](https://kubernetes.io/docs/reference/using-api/deprecation-guide/v1-26). 
 Unsupported: Beta version of the HorizontalPodAutoscaler API Migrate manifests and API clients to use the autoscaling/v2 API version, available since Kubernetes version 1.23. For more information, see [Deprecated API Migration Guide - v1.26](https://kubernetes.io/docs/reference/using-api/deprecation-guide/v1-26). 
 Unsupported: Storage class volume.beta.kubernetes.io/storage-class beta resource annotation Migrate manifests and API clients to use the spec.storageClassName field on the PersistentVolumeClaim and PersistentVolume resources instead. 
 Unsupported: Select Kubernetes API server metrics replaced The following Kubernetes API service metrics were replaced: etcd_db_total_size_in_bytes is replaced by apiserver_storage_db_total_size_in_bytes, job_sync_total is replaced by job_syncs_total, job_finished_total is replaced by jobs_finished_total, and cronjob_job_creation_skew_duration_seconds is replaced by job_creation_skew_duration_seconds. If you rely on these replaced metrics, update accordingly. 







 Update after master 



Changes to make after you update the master to Kubernetes 1.26

 Type Description
For more information, see [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) and [Safely Drain a Node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/).



4. If you notice pod restarts, follow the steps.



1. Get the logs for the failing pod.

kubectl logs --previous -n kube-system <POD NAME>
2. Review the logs and adjust the Ingress resource configurations or the Ingress ConfigMap in the kube-system namespace. For more information, see the NGINX Ingress [Annotations](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/) and [ConfigMap](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/).



5. Wait a few minutes and verify if the failing pods are now running.
6. If the issue persists, contact support. Open a [support case](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar). In the case details, be sure to include any relevant log files, error messages, or command outputs.
Why do pods repeatedly fail to restart or are unexpectedly removed? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

Your pod was healthy but unexpectedly gets removed or gets stuck in a restart loop.

 Why it’s happening 

Your containers might exceed their resource limits, or your pods might be replaced by higher priority pods.

 How to fix it 

See the following sections:



* [Fixing container resource limits](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-failpod-fail-resource-limits)
* [Fixing pod replacement by higher priority pods](https://cloud.ibm.com/docs/openshift?topic=openshift-ts-app-pod-failpod-fail-higher-priority)





 Fixing container resource limits 



1. Get the name of your pod. If you used a label, you can include it to filter your results.

oc get pods --selector='app=wasliberty'
2. Describe the pod and look for the Restart Count.

oc describe pod
3. If the pod restarted many times in a short period of time, fetch its status.

oc get pod <pod_name> -n <namespace> -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"rnLastState: "}}{{.lastState}}{{end}}'
4. Review the reason. For example, OOM Killed means out of memory indicating that the container is crashing because of a resource limit.
5. Add capacity to your cluster such as by [resizing worker pools](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workers) so that the resources can be fulfilled.







 Fixing pod replacement by higher priority pods 

To see if your pod is being replaced by higher priority pods:



1. Get the name of your pod.

oc get pods
2. Describe your pod YAML.
This annotation has been replaced by the kubectl.kubernetes.io/default-container annotation. If your pods rely on the unsupported annotation, update them to use the kubectl.kubernetes.io/default-container annotation instead. For more information, see [Well-Known Labels, Annotations and Taints](https://kubernetes.io/docs/reference/labels-annotations-taints/). 
 Unsupported: Pod seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io annotations Kubernetes no longer fully supports the pod seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io annotations. These annotations have been deprecated since Kubernetes version 1.19 and has been replaced by the securityContext.seccompProfile field for pods and containers. All remaining support for these annotations is planned to be removed in Kubernetes version 1.27. If your pods rely on these unsupported annotations, update them to use the securityContext.seccompProfile field instead. For more information, see [Create Pod that uses the container runtime default seccomp profile](https://kubernetes.io/docs/tutorials/security/seccomp/create-pod-that-uses-the-container-runtime-default-seccomp-profile). 
 Unsupported: Select Kubernetes API server metrics removed The following Kubernetes API service metrics were removed: priority_level_seat_count_watermarks, priority_level_request_count_watermarks and read_vs_write_request_count_watermarks. If you rely on these removed metrics, update accordingly.
Debugging Portworx failures 

Review the options to debug Portworx and find the root causes of any failures.



 Checking whether the pod that mounts your storage instance is successfully deployed 

Follow the steps to review any error messages related to pod deployment.



1. List the pods in your cluster. A pod is successfully deployed if the pod shows a status of Running.

oc get pods
2. Get the details of your pod and review any error messages that are displayed in the Events section of your CLI output.

oc describe pod <pod_name>
3. Retrieve the logs for your pod and review any error messages.

oc logs <pod_name>
4. [Review the Portworx troubleshooting documentation for steps to resolve common errors](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_portworx_storage).







 Restarting your app pod 

Some issues can be resolved by restarting and redeploying your pods. Follow the steps to redeploy a specific pod.



1. If your pod is part of a deployment, delete the pod and let the deployment rebuild it. If your pod is not part of a deployment, delete the pod and reapply your pod configuration file.



1. Delete the pod.

oc delete pod <pod_name>

Example output

pod "nginx" deleted
2. Reapply the configuration file to redeploy the pod.

oc apply -f <app.yaml>

Example output

pod/nginx created



2. If restarting your pod does not resolve the issue, [reload your worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
3. Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.

ibmcloud update

ibmcloud plugin repo-plugins
Why does mounting existing block storage to a pod fail with the wrong file system? 

Virtual Private Cloud

Classic infrastructure

  What’s happening 

When you run oc describe pod <pod_name>, you see the following error:

failed to mount the volume as "ext4", it already contains xfs. Mount error: mount failed: exit status 32

  Why it’s happening 

You have an existing block storage device that is set up with an XFS file system. To mount this device to your pod, you [created a PV](https://cloud.ibm.com/docs/openshift?topic=openshift-block_storageexisting_block) that specified ext4 as your file system or no file system in the spec/flexVolume/fsType section. If no file system is defined, the PV defaults to ext4.

The PV was created successfully and was linked to your existing block storage instance. However, when you try to mount the PV to your cluster by using a matching PVC, the volume fails to mount. You can't mount your XFS block storage instance with an ext4 file system to the pod.

  How to fix it 

Update the file system in the existing PV from ext4 to XFS.



1.  List the existing PVs in your cluster and note the name of the PV that you used for your existing block storage instance.

oc get pv
2.  Save the PV YAML on your local machine.

oc get pv <pv_name> -o yaml > <filepath/xfs_pv.yaml>
3.  Open the YAML file and change the fsType from ext4 to xfs.
4.  Replace the PV in your cluster.

oc replace --force -f <filepath/xfs_pv.yaml>
5.  Log in to the pod where you mounted the PV.

oc exec -it <pod_name> sh
6.  Verify that the file system changed to XFS.

df -Th

Example output:

Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/3600a098031234546d5d4c9876654e35 xfs 20G 33M 20G 1% /myvolumepath