vania-janet/multiturn-rag-retrieval-data huggingface dataset:
In your [cluster dashboard](https://cloud.ibm.com/kubernetes/clusters), click the name of the cluster where you want to install the debug tool add-on.
2. On the Diagnostics and Debug Tool card, click Install.
3. In the dialog box, click Install. Note that it can take a few minutes for the add-on to be installed. To resolve some common issues that you might encounter during the add-on deployment, see [Reviewing add-on state and statuses](https://cloud.ibm.com/docs/containers?topic=containers-debug_addons).
4. On the Diagnostics and Debug Tool card, click Dashboard.
5. In the debug tool dashboard, select the ingress group of tests. Some tests check for potential warnings, errors, or issues, and some tests only gather information that you can reference while you troubleshoot. For more information about the function of each test, click the information icon next to the test's name.
6. Click Run.
7. Check the results of each test.



* If any test fails, click the information icon next to the test's name for information about how to resolve the issue.
* You can also use the results of tests that only gather information while you debug your Ingress service in the following sections.









 Step 3: Check for error messages in your Ingress deployment and the ALB pod logs 

Start by checking for error messages in the Ingress resource deployment events and ALB pod logs. These error messages can help you find the root causes for failures and further debug your Ingress setup in the next sections.



1. Check your Ingress resource deployment and look for warnings or error messages.

kubectl describe ingress <myingress>

In the Events section of the output, you might see warning messages about invalid values in your Ingress resource or in certain annotations that you used.
overlay 102048096 9071556 87786140 10% /
tmpfs 65536 0 65536 0% /dev
tmpfs 7565792 0 7565792 0% /sys/fs/cgroup
shm 65536 0 65536 0% /dev/shm
/dev/vda2 102048096 9071556 87786140 10% /etc/hosts
s3fs 274877906944 0 274877906944 0% /cos-vpc
tmpfs 7565792 44 7565748 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 7565792 0 7565792 0% /proc/acpi
tmpfs 7565792 0 7565792 0% /proc/scsi
tmpfs 7565792 0 7565792 0% /sys/firmware
3. Change directories to the directory where your COS bucket is mounted. In this example the bucket is mounted at /cos-vpc.

cd cos-vpc
4. Write a test.txt file to your COS bucket and list files to verify that the file was written.

touch test.txt && ls
5. Remove the file and log out of your app pod.

rm test.txt && exit
* Make sure that you have a [service access role](https://cloud.ibm.com/docs/openshift?topic=openshift-userschecking-perms) that grants the appropriate Kubernetes RBAC role so that you can work with Kubernetes resources in the project.



To manage rolling updates to your apps,



1. To make sure that your deployments are marked as ready only when the container is running and ready to service requests, add [liveness and readiness probes to your deployment](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_appsprobe).
2. Update your deployment to include a rolling update strategy that specifies the maximum surge and unavailable pods or percentage of pods during the update.

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
replicas: 10
selector:
matchLabels:
service: http-server
minReadySeconds: 5
progressDeadlineSeconds: 600
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 50%
maxSurge: 2
...
Show more

spec.minReadySeconds
: By default, deployments wait until the pod is marked as ready to continue with the rollout. If you notice that the deployment continues to create pods even though your app in the most recent pod is not yet ready, use this field to slow down the deployment rollout. For example, if you specify 5, the deployment waits for 5 seconds after the pod is ready before it creates the next pod.

spec.progressDeadlineSeconds
: Set a timeout in seconds before a deployment is considered failed. For example, without a timeout, if your new app version has a bug and hangs immediately, the rollout can't continue because the pod never reaches a ready state. If you set this timeout to 600 seconds, then if any phase of the rollout fails to progress for 10 minutes, the deployment is marked as failed and the rollout stops.
RabbitMQ gets stuck in a loop after several installation attempts 

After an initial installation or upgrade failure and repeated attempts to retry, the common services RabbitMQ operator pod can get into a CrashLoopBackOff state. For example, the log might include the following types of messages:

"error":"failed to upgrade release: post-upgrade hooks failed: warning:
Hook post-upgrade ibm-rabbitmq/templates/rabbitmq-backup-labeling-job.yaml
failed: jobs.batch "{%name}-ibm-rabbitmq-backup-label" already exists"

Resources for the ibm-rabbitmq-operator.v1.0.11 component must be removed before a new installation or upgrade is started. If too many attempts occur in succession, remaining resources can cause new installations to fail.



1. Delete the RabbitMQ backup label job from the previous installation or upgrade attempt. Look for the name of the job in the logs. The name ends in -ibm-rabbitmq-backup-label:

oc delete job {%name}-ibm-rabbitmq-backup-label -n ${PROJECT_CPD_INSTANCE}
2. Check that the pod returns a Ready state:

oc get pods -n ibm-common-services | grep ibm-rabbitmq







 Preparing to install a size large Watson Assistant deployment 

If you specify large for the watson_assistant_size option when you install Watson Assistant, the installation fails to complete successfully.

Before you install a size large deployment of Watson Assistant, apply the following fix. The following fix uses wa as the name of the Watson Assistant instance and cpd as the namespace where Watson Assistant is installed. These values are set in environment variables. Before you run the command, update the INSTANCE variable with the name of your instance, and update the NAMESPACE variable with the namespace where your instance in installed:

INSTANCE=wa ; NAMESPACE=cpd ; DRY_RUN="--dry-run=client --output=yaml" To apply the changes, change to an empty string
DRY_RUN=""
cat <<EOF | tee wa-network-policy-base.yaml
stork-scheduler-7dd8799cc-knjwt 1/1 Running 0 20h
2. If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, you might not be able to execute all the following commands.



1. Get the names of the containers that run in the driver pod.

kubectl describe pod <pod_name> -n kube-system
2. Export the logs from the driver pod to a logs.txt file on your local machine.

kubectl logs <pod_name> -n kube-system > logs.txt
3. Review the log file.

cat logs.txt



3. Check the latest logs for any error messages. [Review the Portworx troubleshooting documentation for steps to resolve common errors](https://cloud.ibm.com/docs/containers?topic=containers-cs_sitemapsitemap_portworx_storage).







 Checking and updating the kubectl CLI version 

If you use a kubectl CLI version that does not match at least the major.minor version of your cluster, you might experience unexpected results. For example, [Kubernetes does not support](https://kubernetes.io/releases/version-skew-policy/)kubectl client versions that are 2 or more versions apart from the server version (n +/- 2).



1. Verify that the kubectl CLI version that you run on your local machine matches the Kubernetes version that is installed in your cluster. Show the kubectl CLI version that is installed in your cluster and your local machine.

kubectl version

Example output:
: Important: The [26 January 2022 release notes](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-release-notes-datatext-to-speech-data-26january2022) included important notes for the following steps:



* Additional step for performing a mirrored installation of Minio datastore
* Additional steps for performing a mirrored installation of new next-generation models



These additional steps are no longer needed. The case package has been updated to correct the installation issues.





 26 January 2022 (Version 4.0.5) 

Version 4.0.5 is now available
: Text to Speech for IBM Cloud Pak for Data version 4.0.5 is now available. This version supports IBM Cloud Pak for Data version 4.x and Red Hat OpenShift versions 4.6 and 4.8. For more information about installing and managing the service, see [Installing Watson Text to Speech](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=speech-installing-watson-text).

Important: Additional step for performing a mirrored installation of Minio datastore
: Important: These steps are no longer needed if you install case package 4.0.6. For more information, see [28 January 2022 (Version 4.0.5)](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-release-notes-datatext-to-speech-data-28january2022).

If you are performing a mirrored installation (for example, in an air-gapped environment), you need to perform an additional step before completing either of the following steps:



* Step 7 [Mirroring the images to the private container registry](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=registry-mirroring-images-bastion-nodereference_g4h_z1q_tpb__mirror-to-target) of Mirroring images with a bastion model
yq (https://github.com/mikefarah/yq/) version v4.34.1

 kubectl
v1.25.8

 buildctl
buildctl github.com/moby/buildkit v0.11.6 2951a28cd7085eb18979b1f710678623d94ed578

 helm
v3.12.0+gc9f554d

 ibmcloud
ibmcloud version 2.16.0+492aff3-2023-04-06T16:13:44+00:00

 ibmcloud plugins
Listing installed plug-ins...

Plugin Name Version Status Private endpoints supported
cloud-functions[wsk/functions/fn] 1.0.67 false
cloud-internet-services[cis] 1.14.11 true
code-engine[ce] 1.43.5 true
container-registry[cr] 1.0.8 true
container-service[kubernetes-service/ks] 1.0.523 false
cra 1.3.5 false
doi 0.4.2 false
schematics[sch] 1.12.10 true
secrets-manager[sm] 2.0.1 true

 ibmcloud dev
3.0.0

 java
openjdk version "17.0.7" 2023-04-18
IBM Semeru Runtime Open Edition 17.0.7.0 (build 17.0.7+7)
Eclipse OpenJ9 VM 17.0.7.0 (build openj9-0.38.0, JRE 17 Linux amd64-64-Bit Compressed References 20230418_480 (JIT enabled, AOT enabled)
OpenJ9 - d57d05932
OMR - 855813495
JCL - 9d7a231edbc based on jdk-17.0.7+7)

 ant
Apache Ant(TM) version 1.10.13 compiled on January 4 2023

 mvn
Listing installed plug-ins...
Plugin Name Version Status Private endpoints supported
cra 0.1.15 false
doi 0.3.5 false
schematics 1.8.1 true
cloud-functions[wsk/functions/fn] 1.0.58 false
cloud-internet-services[cis] 1.14.2 true
code-engine[ce] 1.29.3 Update Available true
container-registry 0.1.566 true
container-service[kubernetes-service] 1.0.394 false
 ibmcloud dev --version
ibmcloud dev version 2.11.0
 java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Linux amd64-64-Bit Compressed References 20210421_975 (JIT enabled, AOT enabled)
OpenJ9 - b4cc246d9
OMR - 162e6f729
JCL - 7796c80419 based on jdk-11.0.11+9)
 ant -version
Apache Ant(TM) version 1.10.12 compiled on October 13 2021
 mvn -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /opt/IBM/maven
Java version: 11.0.11, vendor: AdoptOpenJDK, runtime: /usr/local/openjdk-11
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.16.18-200.fc35.x86_64", arch: "amd64", family: "unix"
 gradle -version
Listing installed plug-ins...

Plugin Name Version Status Private endpoints supported
cloud-functions[wsk/functions/fn] 1.0.61 false
cloud-internet-services[cis] 1.14.8 true
code-engine[ce] 1.39.3 true
container-registry 1.0.1 true
container-service[kubernetes-service/ks] 1.0.454 false
cra 1.0.2 false
doi 0.3.9 false
schematics 1.12.2 true

 ibmcloud dev --version
ibmcloud dev version 2.14.0

 java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Linux amd64-64-Bit Compressed References 20210421_975 (JIT enabled, AOT enabled)
OpenJ9 - b4cc246d9
OMR - 162e6f729
JCL - 7796c80419 based on jdk-11.0.11+9)

 ant -version
Apache Ant(TM) version 1.10.12 compiled on October 13 2021

 mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /opt/IBM/maven
Java version: 11.0.11, vendor: AdoptOpenJDK, runtime: /usr/local/openjdk-11
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.10.124-linuxkit", arch: "amd64", family: "unix"

 gradle -version

Welcome to Gradle 7.5.1!
* Image tags: 1.11.0-7, 1.16.2-8, 1.17.0-9, 1.18.1-8, and 1.19.0-3.
* Supported cluster versions: 4.3 - 4.6
* Images are now signed.
* Resources that are deployed by the cluster autoscaler add-on are now linked with the corresponding source code and build URLs.
* Updates the Go version to 1.15.5.







 Change log for patch update 1.0.1_146, released 03 December 2020 

Review the changes in version 1.0.1_146 of the cluster autoscaler add-on.



* Image tags: 1.15.4-4, 1.16.2-7, 1.17.0-8, 1.18.1-7, and 1.19.0-2.
* Supported cluster versions: 4.3 - 4.6
* Updates the cluster autoscaler to run as non-root.
* Adds a feature to validate secrets before the autoscaler pods are initialized.







 Change log for patch update 1.0.1_128, released 27 October 2020 

Review the changes in version 1.0.1_128 of the cluster autoscaler add-on.



* Image tags: 1.15.4-4, 1.16.2-6, 1.17.0-7, 1.18.1-6, and 1.19.0-1.
* Supported cluster versions: 4.3 - 4.5
* Updates the Go version to 1.15.2.







 Change log for patch update 1.0.1_124, released 16 October 2020 

Review the changes in version 1.0.1_124 of the cluster autoscaler add-on.



* Image tags: 1.15.4-4, 1.16.2-6, 1.17.0-7, 1.18.1-6, and 1.19.0-1.
* Supported cluster versions: 4.3 - 4.5
* Exposes the --new-pod-scale-up-delay option in the ConfigMap.
: Important: With Text to Speech for IBM Cloud Pak for Data version 4.0.3, PostgreSQL pods based on the EnterpriseDB version 1.10 operator can fail to start. This prevents the Speech services from starting. A workaround exists for this problem. If your Speech services fail to start, see [PostgreSQL pods do not start with EnterpriseDB version 1.10 operator](https://www.ibm.com/support/pages/node/6525340) for information about diagnosing and resolving the problem.

This problem is fixed in Text to Speech for IBM Cloud Pak for Data version 4.0.4.

New support for IBM Spectrum Scale Container Native storage class
: Since version 4.0.3, the Speech services support the IBM Spectrum® Scale Container Native storage class. To use IBM Spectrum Scale, specify "ibm-spectrum-scale-sc" for the storageClass property of the Speech services custom resource. For more information, see [Installing Watson Text to Speech](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=speech-installing-watson-text).

Interaction of Speech services with MinIO datastore during installation
: The Speech services runtime components, sttRuntime and ttsRuntime, cannot start until the models and voices for the services are fully uploaded into the MinIO datastore. During installation, the services might fail and automatically restart themselves one or more times until upload of the models and voices is complete. They then start properly. No user action is required.

Defect fix: Improve upgrade documentation
: Defect fix: Documentation for upgrading the Speech services to new versions of IBM Cloud Pak for Data version 4.0.x included incorrect references in some commands. These references are now correct:



* The strings watsonSpeechToTextStatus and watsonTextToSpeechStatus have been changed to speechStatus in both cases.
* The strings status.watsonSpeechToTextVersion and status.watsonTextToSpeechVersion have been changed to .spec.version in both cases.
* [Customizing security context constraints](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_scccustomize_sccs)
* [Default Red Hat OpenShift security context constraints](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_sccoc_sccs)
* [Default IBM security context constraints](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_sccibm_sccs)



[Setting pod priority](https://cloud.ibm.com/docs/openshift?topic=openshift-pod_prioritypod_priority)



* [Understanding default priority classes](https://cloud.ibm.com/docs/openshift?topic=openshift-pod_prioritydefault_priority_class)
* [Creating a priority class](https://cloud.ibm.com/docs/openshift?topic=openshift-pod_prioritycreate_priority_class)
* [Assigning priority to your pods](https://cloud.ibm.com/docs/openshift?topic=openshift-pod_priorityprioritize)



[Setting Kubernetes API priority and fairness](https://cloud.ibm.com/docs/openshift?topic=openshift-kubeapi-prioritykubeapi-priority)



* [Reviewing default flow schema and priority levels](https://cloud.ibm.com/docs/openshift?topic=openshift-kubeapi-prioritykubeapi-default-priority)



* [Viewing Red Hat OpenShift on IBM Cloud created priority level configurations](https://cloud.ibm.com/docs/openshift?topic=openshift-kubeapi-prioritykube-api-prioritylevelconfig)





[Pod security admission](https://cloud.ibm.com/docs/openshift?topic=openshift-pod-security-admissionpod-security-admission)
To address the issue, first determine which pod is out of memory by completing the following steps:

If the document status cannot be promoted to Processing, check the status of the inlet, outlet, and converter pods.



1. Run the following command:

oc get pod -l 'tenant=wd,run in (inlet,outlet,converter)'
2. If any of the pods are not showing a Running status, restart the failing pod by using the following command:

oc delete pod <pod_name>
3. Otherwise, open the Manage collections>{collection name}>Activity page. Check the Warnings and errors at a glance section for the message, OutOfMemory happened during conversion. Please reconsider size of documents. If shown, use the following command to raise the memory size for the converter:

oc patch wd wd --type=merge --patch='{"spec":{"ingestion":{"converter":b{"maxHeapMemory":"10240m","resources":{"limits":{"memory":"10Gi"}}}}}}'

Adjust the value of maxHeapMemory and the container memory according to your cluster resources.
4. After the converter pod restarts successfully, click Reprocess from the Activity tab.



If documents are stuck in Processing status and cannot be promoted to the Available status for a collection, complete the following steps:



1. Check the status of Hadoop by using the following command:

oc get pod -l 'tenant=wd,run in (hdp-rm,hdp-worker)'

where l is a lowercase L for list.
2. If any of the pods are not showing a Running status, restart the failing pod by using the following command:

oc delete pod <pod_name>
3. Check whether any of the Hadoop worker nodes has insufficient memory by using the following command to look for the OOM when allocating message:
Blindly installing a smart contract package from another organization risks overwriting any customizations that you might have added to your version of the smart contract.





 Important considerations when you update smart contracts 



1. What happens to my peers if I forget to upgrade them to the latest version before committing the smart contract definition on the channel?

After updating a smart contract definition to use a use smart contract package and committing it to the channel, if there are still peers on the channel running the previous version, those peers are no longer able to endorse transactions for the smart contract. Also, you risk not having enough endorsements for transactions to be committed to the ledger, depending on how the smart contract endorsement policy is defined. However, it is possible to install the new version of the smart contract package on these peers after the smart contract definition is committed to the channel and the peers will again be able to endorse transactions, effectively catching up.
2. What happens when I remove an organization from my private data collection?

The peers in that organization continue to store data in the private data collection until its ledger reaches the block that removes its membership from the collection. After that occurs, the peers will not receive private data in any future transactions, and clients of that organization will no longer be able to query the private data via a smart contract from any peer.
3. What happens to the existing chaincode-execution pod after I update my smart contract?

Whenever a transaction is submitted to a smart contract installed on peer, a chaincode-execution pod is launched if one does not already exist. When you deploy a new version of your smart contract, the chaincode-execution pod for the previous version of the smart contract continues to run but is no longer required. To save cluster resources, you can manually delete the chaincode-execution pods for the prior smart contract versions. From your cluster namespace, run the following command to see the list of chaincode-execution pods that are currently running:

kubectl get pod -n <NAMESPACE> | grep chaincode-execution | cut -d" " -f1 | xargs -I {} kubectl get po {} -n <NAMESPACE> --show-labels

Replace <NAMESPACE> with the name of your cluster namespace.
If restarting your pod does not resolve the issue, [reload your worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
3. Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.

ibmcloud update

ibmcloud plugin repo-plugins

ibmcloud plugin update







 Verifying that the storage driver and plug-in pods show a status of Running 

Follow the steps to check the status of your storage driver and plug-in pods and review any error messages.



1. List the pods in the kube-system project.

oc get pods -n kube-system
2. If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, the following commands might fail.



1. Get the names of the containers that run in the driver pod.

kubectl describe pod <pod_name> -n kube-system
2. Export the logs from the driver pod to a logs.txt file on your local machine.

oc logs <pod_name> -n kube-system > logs.txt
3. Review the log file.

cat logs.txt



3. Check the latest logs for any error messages. [Review the ODF troubleshooting documentation for steps to resolve common errors](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_openshift_data_foundation).







 Checking and updating the oc CLI version 

If you use a oc CLI version that does not match at least the major.minor version of your cluster, you might experience unexpected results.
OS name: "linux", version: "4.9.184-linuxkit", arch: "amd64", family: "unix"

 gradle -version

Welcome to Gradle 5.4.1!

Here are the highlights of this release:
- Run builds with JDK12
- New API for Incremental Tasks
- Updates to native projects, including Swift 5 support

For more details see https://docs.gradle.org/5.4.1/release-notes.html

------------------------------------------------------------
Gradle 5.4.1
------------------------------------------------------------

Build time: 2019-04-26 08:14:42 UTC
Revision: 261d171646b36a6a28d5a19a69676cd098a4c19d

Kotlin: 1.3.21
Groovy: 2.5.4
Ant: Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM: 1.8.0_212 (Oracle Corporation 25.212-b03)
OS: Linux 4.9.184-linuxkit amd64
Show more





 Version 2.2 

To view the contents of version 2.2, from the running image, type default_versions.sh. This image includes the following tools:

 node --version
v10.16.0

 npm --version
6.9.0

 jq --version
jq-1.6

 yq --version
yq version 2.4.0

 kubectl version --client