vania-janet/multiturn-rag-retrieval-data huggingface dataset:
error "$(date +"%b %d %G %H:%M:%S"): Unable to fetch controller pod."
if [ "$controllerPodName" == "" ]]; then
echo "$(date +"%b %d %G %H:%M:%S"): VPC Block CSI Driver addon is not enabled"
exit 1
fi

echo "apiVersion: v1
data:
ibm-credentials.env: $encodedValue
kind: Secret
metadata:
name: ibm-cloud-credentials
namespace: kube-system
type: Opaque" > ibm-cloud-credentials.yaml

create the k8s secret
kubectl apply -f ibm-cloud-credentials.yaml &> /dev/null
error "$(date +"%b %d %G %H:%M:%S"): Error creating ibm-cloud-credentials secret."
echo "$(date +"%b %d %G %H:%M:%S"): Created ibm-cloud-credentials secret"

restart the controller pod
echo "$(date +"%b %d %G %H:%M:%S"): Restarting $controllerPodName pod"
kubectl delete pod $controllerPodName -n kube-system &> /dev/null
error "$(date +"%b %d %G %H:%M:%S"): Error restarting $controllerPodName pod in kube-system namespace."

controllerPodStatus=
for i in {1..12}
do
sleep 5
controllerPodStatus=$(kubectl get pods -n kube-system | grep ibm-vpc-block-csi-controller | awk '{print $3}')
if [ "$controllerPodStatus" == "Running" ]]; then
echo "$(date +"%b %d %G %H:%M:%S"): VPC Block CSI Driver is now using ibm-cloud-credentials secret"
Review the pod details to verify that the plug-in installation succeeded.



1. Verify the installation succeeded by listing the driver pods.

oc get pod --all-namespaces -o wide | grep object

Example output

ibmcloud-object-storage-driver-9n8g8 1/1 Running 0 2m
ibmcloud-object-storage-plugin-7c774d484b-pcnnx 1/1 Running 0 2m

The installation is successful when you see one ibmcloud-object-storage-plugin pod and one or more ibmcloud-object-storage-driver pods. The number of ibmcloud-object-storage-driver pods equals the number of worker nodes in your cluster. All pods must be in a Running state for the plug-in to function properly. If the pods fail, run oc describe pod -n ibm-object-s3fs <pod_name> to find the root cause for the failure.
2. Verify that the storage classes are created successfully.

oc get storageclass | grep s3

Example output

ibmc-s3fs-cold-cross-region ibm.io/ibmc-s3fs 8m
ibmc-s3fs-cold-regional ibm.io/ibmc-s3fs 8m
ibmc-s3fs-flex-cross-region ibm.io/ibmc-s3fs 8m
ibmc-s3fs-flex-perf-cross-region ibm.io/ibmc-s3fs 8m
ibmc-s3fs-flex-perf-regional ibm.io/ibmc-s3fs 8m
ibmc-s3fs-flex-regional ibm.io/ibmc-s3fs 8m
ibmc-s3fs-standard-cross-region ibm.io/ibmc-s3fs 8m
ibmc-s3fs-standard-perf-cross-region ibm.io/ibmc-s3fs 8m
ibmc-s3fs-standard-perf-regional ibm.io/ibmc-s3fs 8m
timeout: 3600 
 Additional parameters: -e hana.memoryRequest=7Gi -e storageGateway.replicas=2 -e vsystem.vRep.exportsMask=true 
 Cluster Name: sap-di32-cluster 
 S-User Name: Sxxxxxxxx 
 S-User Password: xxxxxxxx 



The installation and the initialization of all pods may take a while to complete.



1. You may want to watch how the pods get initialized and start running

watch oc get -n sdi pods -o wide





 Final Step 

[Testing your installation](https://help.sap.com/viewer/a8d90a56d61a49718ebcb5f65014bbe7/3.1.latest/en-US/1551785f3d7e4d37af7fe99185f7acb6.html).
Debugging OpenShift Data Foundation failures 

Review the options to debug ODF and find the root causes of any failures.



 Checking whether the pod that mounts your storage instance is successfully deployed 

Follow the steps to review any error messages related to pod deployment.



1. List the pods in your cluster. A pod is successfully deployed if the pod shows a status of Running.

oc get pods
2. Get the details of your pod and review any error messages that are displayed in the Events section of your CLI output.

oc describe pod <pod_name>
3. Retrieve the logs for your pod and review any error messages.

oc logs <pod_name>
4. [Review the ODF troubleshooting documentation for steps to resolve common errors](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_openshift_data_foundation).







 Restarting your app pod 

Some issues can be resolved by restarting and redeploying your pods. Follow the steps to redeploy a specific pod.



1. If your pod is part of a deployment, delete the pod and let the deployment rebuild it. If your pod is not part of a deployment, delete the pod and reapply your pod configuration file.



1. Delete the pod.

oc delete pod <pod_name>

Example output

pod "nginx" deleted
2. Reapply the configuration file to redeploy the pod.

oc apply -f <app.yaml>

Example output

pod/nginx created



2. If restarting your pod does not resolve the issue, [reload your worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
3. Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.

ibmcloud update

ibmcloud plugin repo-plugins
Why can't my app access or write to PVCs? 

Virtual Private Cloud

Classic infrastructure

When you mount a PVC to your pod, you might experience errors when accessing or writing to the PVC.



1.  List the pods in your cluster and review the status of the pod.

oc get pods
2.  Find the root cause for why your app can't access or write to the PVC.

oc describe pod <pod_name>

oc logs <pod_name>
3.  Review common errors that can occur when you mount a PVC to a pod.





Block Storage error messages

 Symptom or error message                                                                                                       Description                                                                                                                                                                                                                                                                                                          Steps to resolve                                                                                                                                                                             

 Your pod is stuck in a ContainerCreating or CrashLoopBackOff state. MountVolume.SetUp failed for volume ... read-only.         The IBM Cloud infrastructure back end experienced network problems. To protect your data and to avoid data corruption, IBM Cloud automatically disconnected the block storage server to prevent write operations on block storage instances.                                                                         See [Block storage: Block storage changes to read-only](https://cloud.ibm.com/docs/openshift?topic=openshift-readonly_block)                                                                                  
 failed to mount the volume as "ext4", it already contains xfs. Mount error: mount failed: exit status 32.                      You want to mount an existing block storage instance that was set up with an XFS file system. When you created the PV and the matching PVC, you specified an ext4 or no file system. The file system that you specify in your PV must be the same file system that is set up in your block storage instance.         See [Block storage: Mounting existing block storage to a pod fails due to the wrong file system](https://cloud.ibm.com/docs/openshift?topic=openshift-block_filesystem)
However, some pods attempt to parse the variable, which results in the error.
* Solution: [Install patch cpd-watson-discovery-2.2.1-patch-1](https://www.ibm.com/docs/en/cloud-paks/cp-data/3.5.0?topic=iwd-installing-watson-discoverysvc-install__patches-section), which fixes this issue.





Also, see the issues in all previous releases.





 2.2, 8 December 2020 



* When a small CSV file (generally a CSV with 99 lines or fewer) is uploaded, the header and/or first row may not be ingested correctly. If this happens, in the tooling, navigate to the CSV Settings tab and update the settings. After reprocessing, navigate to the Manage fields tab and update the field types if needed.
* If you have set up your collections using a custom crawler built with the [IBM Cloud Pak for Data custom connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-build-connector), and then remove the custom crawler deployment, the Processing Settings page will not display the crawler configuration. This is because the underlying crawler is not available. To work around this issue, confirm that the custom crawler is deployed when there are collections using it.
* When using a [IBM Cloud Pak for Data custom connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-build-connector) with Discovery for IBM Cloud Pak for Data 2.2, the script scripts/manage_custom_crawler.sh used to deploy and remove the deployment of the custom crawler fails. To work around this issue, replace line 37 podname="gateway" with podname="wd-discovery-gateway" in scripts/manage_custom_crawler.sh, and then rerun the deploy command.
* When you create a custom enrichment in the tooling, you must choose a field the enrichment should be applied to and click Apply. If no field is selected, then the Apply and reprocess button will be disabled for enrichments changes until the new enrichment has a field.
Java version: 11.0.10, vendor: AdoptOpenJDK, runtime: /usr/local/openjdk-11
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.10.42", arch: "amd64", family: "unix"

 gradle -version

Welcome to Gradle 7.0.2!

Here are the highlights of this release:
- File system watching enabled by default
- Support for running with and building Java 16 projects
- Native support for Apple Silicon processors
- Dependency catalog feature preview

For more details see https://docs.gradle.org/7.0.2/release-notes.html

------------------------------------------------------------
Gradle 7.0.2
------------------------------------------------------------

Build time: 2021-05-14 12:02:31 UTC
Revision: 1ef1b260d39daacbf9357f9d8594a8a743e2152e

Kotlin: 1.4.31
Groovy: 3.0.7
Ant: Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM: 11.0.10 (Eclipse OpenJ9 openj9-0.24.0)
OS: Linux 5.10.42 amd64

 oc version
Unable to connect to the server: dial tcp 172.21.0.1:443: i/o timeout
Client Version: 4.7.13

 zip
* [Fixing pod replacement by higher priority pods](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-failpod-fail-higher-priority)



[Why does binding a service to a cluster result in a same name error?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-svc-bind-namets-app-svc-bind-name)

[Why does binding a service to a cluster results in service not found error?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-svc-bind-not-foundts-app-svc-bind-not-found)



* [Cluster user](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-svc-bind-not-foundservice-bind-cluster-user)
* [As the account admin](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-svc-bind-not-foundservice-bind-account-admin)



[Why does binding a service to a cluster results in service does not support service keys error?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-svc-keyts-app-svc-key)

[Why can't I install a Helm chart with updated configuration values?](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-helm-installts-app-helm-install)





 Managed add-ons 

[Debugging cluster add-ons](https://cloud.ibm.com/docs/containers?topic=containers-debug_addonsdebug_addons)

[Debugging Istio](https://cloud.ibm.com/docs/containers?topic=containers-istio_debug_toolistio_debug_tool)
- name: CERTIFICATES_IMPORT_LIST
value: /etc/secrets/kafka/ca.pem:kafka_ca,/etc/secrets/custom/ca.pem:custom_ca
volumeMounts:
- mountPath: /etc/secrets/custom
name: custom-cert
readOnly: true
volumes:
- name: custom-cert
secret:
defaultMode: 420
items:
- key: ca_cert
path: ca.pem
secretName: ${INSTANCE}-custom-webhooks-cert
EOF
Show more
3. Wait approximately 10 minutes for the wa-webhooks-connector pod to restart. This pod restarts automatically.
4. After the pod restarts, check the logs by running the following command. In the command, replace XXXX with the suffix of the wa-webhooks-connector pod:

oc logs wa-webhooks-connector-XXXX // Replace XXXX with the suffix of the wa-webhooks-connector pod

After you run this command, you should see two lines similar to the following example at the beginning of the log:

Certificate was added to keystore
Certificate was added to keystore

When you see these two lines, then the custom certificate was properly imported into the keystore.









 4.0.5 



 Install Redis with Watson Assistant if foundational services version is higher than 3.14.1 

If you are installing the Redis operator with Watson Assistant for IBM Cloud Pak® for Data 4.0.5 with an IBM Cloud Pak foundational services version higher than 3.14.1, the Redis operator might get stuck in Pending status. If you have an air-gapped cluster, complete the steps in the [Air-gapped cluster](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-troubleshoottroubleshoot-405-install-redis-foundational-services-air-gapped) section to resolve this issue.
: When you upgrade the Speech services from version 4.0.x to version 4.5.x, you might encounter an issue where the PostgreSQL pods become stuck in the Terminating state. If this problem occurs during your upgrade, perform the following steps to resolve the problem. The information and steps are also documented in Upgrading Watson Speech services from Version 4.0 to Version 4.5 in the Upgrading topic of [Watson Speech services on IBM Cloud Pak for Data](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=services-watson-speech).



1. Use the following command to identify pods that remain in the Terminating state:



oc get pods -n ${PROJECT_CPD_INSTANCE} -o wide | awk {'print $1'}



1. Use the following command to set the environment variable pods to include the list of pods that remain in the Terminating state:



pods=$(oc get pods -n ${PROJECT_CPD_INSTANCE} -o wide | awk {'print $1'})



1. Use the following command to delete the stuck pods so that the upgrade process can continue:



pods=$(oc get pods -n ${PROJECT_CPD_INSTANCE} -o wide | grep Terminating | awk {'print $1'})

Defect fix: Fix custom resource entries documentation
: Defect fix: The documentation for the Speech services custom resource now includes colons after the names of the models koKrTelephony and nlNlTelephony. Previously, the documentation for these two entries omitted the colons.

Security vulnerabilities addressed
: The following security vulnerabilities have been fixed:



* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a buffer over-read flaw in Linux Kernel (CVE-2020-28915)](https://www.ibm.com/support/pages/node/6829133)
Why can't my app access or write to PVCs? 

Virtual Private Cloud

Classic infrastructure

When you mount a PVC to your pod, you might experience errors when accessing or writing to the PVC.



1.  List the pods in your cluster and review the status of the pod.

kubectl get pods
2.  Find the root cause for why your app can't access or write to the PVC.

kubectl describe pod <pod_name>

kubectl logs <pod_name>
3.  Review common errors that can occur when you mount a PVC to a pod.





Block Storage error messages

 Symptom or error message                                                                                                       Description                                                                                                                                                                                                                                                                                                          Steps to resolve                                                                                                                                                                             

 Your pod is stuck in a ContainerCreating or CrashLoopBackOff state. MountVolume.SetUp failed for volume ... read-only.         The IBM Cloud infrastructure back end experienced network problems. To protect your data and to avoid data corruption, IBM Cloud automatically disconnected the block storage server to prevent write operations on block storage instances.                                                                         See [Block storage: Block storage changes to read-only](https://cloud.ibm.com/docs/containers?topic=containers-readonly_block)                                                                                  
 failed to mount the volume as "ext4", it already contains xfs. Mount error: mount failed: exit status 32.                      You want to mount an existing block storage instance that was set up with an XFS file system. When you created the PV and the matching PVC, you specified an ext4 or no file system. The file system that you specify in your PV must be the same file system that is set up in your block storage instance.         See [Block storage: Mounting existing block storage to a pod fails due to the wrong file system](https://cloud.ibm.com/docs/containers?topic=containers-block_filesystem)
Next, you can include your apps in the [Istio service mesh](https://cloud.ibm.com/docs/containers?topic=containers-istio-meshistio_sidecar).









 Installing the istioctl CLI 

Install the istioctl CLI client. For more information, see the [istioctl command reference](https://istio.io/latest/docs/reference/commands/istioctl/).



1. Check the version of Istio that you installed.

istioctl version
2. Download the version of istioctl that matches your cluster's Istio version.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.18.0 sh -
3. Navigate to the Istio package directory.

cd istio-1.18.0
4. Linux and macOS users: Add the istioctl client to your PATH system variable.

export PATH=$PWD/bin:$PATH







 Customizing the Istio installation 

You can customize a set of Istio configuration options by editing the managed-istio-custom ConfigMap resource. These settings include extra control over monitoring, logging, and networking in your control plane and service mesh.



1. Describe the managed-istio-custom ConfigMap resource to review its contents and the in-line documentation.

kubectl describe cm managed-istio-custom -n ibm-operators
2. Edit the managed-istio-custom ConfigMap resource.

kubectl edit cm managed-istio-custom -n ibm-operators
3. In the data section, add the <key>: "<value>" pair of one or more of the following configuration options.

istio-components-pilot-requests-cpu
: Default value: "500m"
: Configure the CPU request in milli CPU for the istiod component pod. Use caution when changing this value.
Then, verify the pods are ready by listing the pods in the openshift-storage namespace.

oc get pods -n openshift-storage

Example output

NAME READY STATUS RESTARTS AGE
ocs-metrics-exporter-5b85d48d66-lwzfn 1/1 Running 0 2d1h
ocs-operator-86498bf74c-qcgvh 1/1 Running 0 2d1h
odf-console-68bcd54c7c-5fvkq 1/1 Running 0 2d1h
rook-ceph-mgr-a-758845d77c-xjqkg 2/2 Running 0 2d1h
rook-ceph-mon-a-85d65d9f66-crrhb 2/2 Running 0 2d1h
rook-ceph-mon-b-74fd78856d-s2pdf 2/2 Running 0 2d1h
rook-ceph-mon-c-76f9b8b5f9-gqcm4 2/2 Running 0 2d1h
rook-ceph-operator-5d659cb494-ctkx6 1/1 Running 0 2d1h
rook-ceph-osd-0-846cf86f79-z97mc 2/2 Running 0 2d1h
rook-ceph-osd-1-7f79ccf77d-8g4cn 2/2 Running 0 2d1h
rook-ceph-osd-2-549cc486b4-7wf5k 2/2 Running 0 2d1h
rook-ceph-osd-prepare-ocs-deviceset-0-data-0z6pn9-6fwqr 0/1 Completed 0 10d
rook-ceph-osd-prepare-ocs-deviceset-1-data-0kkxrw-cppk9 0/1 Completed 0 10d
Debugging Portworx failures 

Review the options to debug Portworx and find the root causes of any failures.



 Checking whether the pod that mounts your storage instance is successfully deployed 

Follow the steps to review any error messages related to pod deployment.



1. List the pods in your cluster. A pod is successfully deployed if the pod shows a status of Running.

kubectl get pods
2. Get the details of your pod and review any error messages that are displayed in the Events section of your CLI output.

kubectl describe pod <pod_name>
3. Retrieve the logs for your pod and review any error messages.

kubectl logs <pod_name>
4. [Review the Portworx troubleshooting documentation for steps to resolve common errors](https://cloud.ibm.com/docs/containers?topic=containers-cs_sitemapsitemap_portworx_storage).







 Restarting your app pod 

Some issues can be resolved by restarting and redeploying your pods. Follow the steps to redeploy a specific pod.



1. If your pod is part of a deployment, delete the pod and let the deployment rebuild it. If your pod is not part of a deployment, delete the pod and reapply your pod configuration file.



1. Delete the pod.

kubectl delete pod <pod_name>

Example output

pod "nginx" deleted
2. Reapply the configuration file to redeploy the pod.

kubectl apply -f <app.yaml>

Example output

pod/nginx created



2. If restarting your pod does not resolve the issue, [reload your worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reload).
3. Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.

ibmcloud update

ibmcloud plugin repo-plugins
After you try to install support for private workers in Kubernetes, an error message is displayed to indicate that there is a schema mismatch and the installation fails.

SchemaError(io.k8s.apimachinery.pkg.apis.meta.v1.APIGroup): invalid object doesn't have additional properties

 Why it’s happening 

There is a mismatch between the versions of kubectl that you are running on the Kubernetes server and the Kubernetes client.

 How to fix it 

Install the latest version of kubectl on the client machine.





 Why can't I pull images for tekton-releases or pipeline-private-worker from some container registries? 

Cluster security prevents you from pulling down images.

 What’s happening 

When you try to install the private worker framework, an error message is displayed.

Error from server (InternalError): error when creating "https://private-worker-service.us-south.devops.cloud.ibm.com/install": Internal error occurred: admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
Deny "gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller@sha256:80e040a58ce6c4d58ae893eb934777bce013ef8be079967dc3db783d76fa5aaa", no matching repositories in ClusterImagePolicy and no ImagePolicies in the "tekton-pipelines" namespace
Error from server (InternalError): error when creating "https://private-worker-service.us-south.devops.cloud.ibm.com/install": Internal error occurred: admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
stork-scheduler-7dd8799cc-knjwt 1/1 Running 0 20h
2. If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, you might not be able to execute all the following commands.



1. Get the names of the containers that run in the driver pod.

kubectl describe pod <pod_name> -n kube-system
2. Export the logs from the driver pod to a logs.txt file on your local machine.

oc logs <pod_name> -n kube-system > logs.txt
3. Review the log file.

cat logs.txt



3. Check the latest logs for any error messages. [Review the Portworx troubleshooting documentation for steps to resolve common errors](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_portworx_storage).







 Checking and updating the oc CLI version 

If you use a oc CLI version that does not match at least the major.minor version of your cluster, you might experience unexpected results. For example, [Kubernetes does not support](https://kubernetes.io/releases/version-skew-policy/)oc client versions that are 2 or more versions apart from the server version (n +/- 2).



1. Verify that the oc CLI version that you run on your local machine matches the Kubernetes version that is installed in your cluster. Show the oc CLI version that is installed in your cluster and your local machine.

oc version

Example output: