vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The SDK uses the persistent cache to store the App Configuration data that is available across your application restarts.

// 1. default (without persistent cache)
appConfiguration.SetContext(collectionId, environmentId)
// 2. with persistent cache
appConfiguration.SetContext(collectionId, environmentId, AppConfiguration.ContextOptions{
PersistentCacheDirectory: "/var/lib/docker/volumes/",
})

Where persistentCacheDirectory: Absolute path to a directory that has read and write permission for the user. The SDK creates a file AppConfiguration.json in the specified directory, and it is used as the persistent cache to store the App Configuration service information.

When persistent cache is enabled, the SDK keeps the last known good configuration at the persistent cache. If the App Configuration server is unreachable, the latest configurations at the persistent cache are loaded to the application to continue working.

Make sure that the cache file is not lost or deleted in any case. For example, consider the case when a kubernetes pod is restarted and the cache file (appconfiguration.json) was stored in ephemeral volume of the pod. As pod gets restarted, kubernetes destroys the ephermal volume in the pod, as a result the cache file gets deleted. So, make sure that the cache file created by the SDK is always stored in persistent volume by providing the correct absolute path of the persistent directory.





 Offline options 

The SDK is also designed to serve configurations, and perform feature flag and property evaluations without being connected to App Configuration service.

appConfiguration.SetContext(collectionId, environmentId, AppConfiguration.ContextOptions{
BootstrapFile: "saflights/flights.json",
LiveConfigUpdateEnabled: false,
})

Where,



* BootstrapFile: Absolute path of the JSON file, which contains configuration details. Make sure to provide a proper JSON file.
$ cocoa inventory promote --org='Github-ID' --repo='compliance-inventory-repo' --source='master' --target='prod' --priority='Critical' --assigned-to='employee@ibm.com' --description='desc' --purpose='purpose' --impact='impact' --customer-impact='impact' --backout-plan='rollback'







 cocoa incident commands 



 cocoa incident add 

Creates or updates an incident issue for a failing task in a pipeline run based on the commit-hash.

Options:

--task (Required) The name of the failing task
--commit-hash (Required) The commit hash which triggered the pipeline run
--pipeline-run-url The pipeline run url where the task ran
--org The incident issue repo org
--repo The incident issue repo name
--assignee (Optional) Assignee(s) for the incident issue (github username) e.g: --assigne=jane-doe --assignee=john-smith
--app-repo-url The app repo URL
--label Label(s) to add to the incident issue (optional) e.g: --label=foo --label=bar
--git-token-path (Optional) Github Token's path
--git-api-url (Optional) Github API url

Required Environment Variables:

GHE_ORG= Can be used instead of --org (either the option or the variable is required)
GHE_REPO= Can be used instead of --repo (either the option or the variable is required)
PIPELINE_RUN_URL= Can be used instead of --pipeline-run-url (either the option or the variable is required)
APP_REPO_URL= Can be used instead of --app-repo-url (either the option or the variable is required)

Required Environment Variables, if you are using GitHub:
2. Get the logs for the pod you want to troubleshoot.

oc logs POD -n NAMESPACE







 Review the details 

Review the error messages, logs, and details of your configurations and troubleshoot. Review the [troubleshooting documentation](https://cloud.ibm.com/docs/satellite?topic=satellite-sitemapsitemap_storage) for further debugging steps.





 Contact Support 

If the issue persists, open a [support case](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar). In the case details, be sure to include any relevant log files, error messages, or command outputs.
Why does mounting existing block storage to a pod fail with the wrong file system? 

Virtual Private Cloud

Classic infrastructure

  What’s happening 

When you run kubectl describe pod <pod_name>, you see the following error:

failed to mount the volume as "ext4", it already contains xfs. Mount error: mount failed: exit status 32

  Why it’s happening 

You have an existing block storage device that is set up with an XFS file system. To mount this device to your pod, you [created a PV](https://cloud.ibm.com/docs/containers?topic=containers-block_storageexisting_block) that specified ext4 as your file system or no file system in the spec/flexVolume/fsType section. If no file system is defined, the PV defaults to ext4.

The PV was created successfully and was linked to your existing block storage instance. However, when you try to mount the PV to your cluster by using a matching PVC, the volume fails to mount. You can't mount your XFS block storage instance with an ext4 file system to the pod.

  How to fix it 

Update the file system in the existing PV from ext4 to XFS.



1.  List the existing PVs in your cluster and note the name of the PV that you used for your existing block storage instance.

kubectl get pv
2.  Save the PV YAML on your local machine.

kubectl get pv <pv_name> -o yaml > <filepath/xfs_pv.yaml>
3.  Open the YAML file and change the fsType from ext4 to xfs.
4.  Replace the PV in your cluster.

kubectl replace --force -f <filepath/xfs_pv.yaml>
5.  Log in to the pod where you mounted the PV.

kubectl exec -it <pod_name> sh
6.  Verify that the file system changed to XFS.

df -Th

Example output:

Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/3600a098031234546d5d4c9876654e35 xfs 20G 33M 20G 1% /myvolumepath
The following errors are displayed during the developer toolkit installation:

sqlite3@3.1.1 install /usr/local/lib/node_modules/strong-pm/node_modules/minkelite/node_modules/sqlite3
node-pre-gyp install --fallback-to-build
/usr/bin/env: node: No such file or directory
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read /usr/share/doc/nodejs/README.Debian
npm ERR! weird error 127
npm ERR! not ok code 0

 How to fix it 

Enter the following command to resolve the problem:

$ update-alternatives --install /usr/bin/node node /usr/bin/nodejs 99





 Unable to debug the npm installation failure 

When you follow the steps to install the developer toolkit, the npm installation fails.

 What’s happening 

The npm installation fails without providing any useful information to debug.

 How to fix it 

When an installation fails, npm writes a line in the npm-debug.log</filepath> file to show where the error is located. Use the npm-debug.log file to determine the cause.





 Unable to open the API Designer 

You enter the command apic edit and the API Designer does not open.

 What’s happening 

You are unable to open an instance of the API Designer after you enter the command:

apic edit

and the following message is displayed:

<time stamp>. 329Z ERROR apiConnect: listen EADDRINUSE <ip address> :9000

 Why it’s happening 

You have already started an instance of the API Designer from another command window.

 How to fix it 

To fix this problem, you need to close the other command window as described in the following steps:



1. In the other command window, press Ctrl + C.
2. The following message is displayed.



Terminate Batch job (Y/N)?



3.
If restarting your pod does not resolve the issue, [reload your worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
3. Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.

ibmcloud update

ibmcloud plugin repo-plugins

ibmcloud plugin update







 Verifying that the Portworx storage driver and plug-in pods show a status of Running 

Follow the steps to check the status of your storage driver and plug-in pods and review any error messages.



1. List the pods in the kube-system project.

oc get pods -n kube-system | grep portworx|stork

Example output:

portworx-594rw 1/1 Running 0 20h
portworx-rn6wk 1/1 Running 0 20h
portworx-rx9vf 1/1 Running 0 20h
stork-6b99cf5579-5q6x4 1/1 Running 0 20h
stork-6b99cf5579-slqlr 1/1 Running 0 20h
stork-6b99cf5579-vz9j4 1/1 Running 0 20h
stork-scheduler-7dd8799cc-bl75b 1/1 Running 0 20h
stork-scheduler-7dd8799cc-j4rc9 1/1 Running 0 20h
stork-scheduler-7dd8799cc-knjwt 1/1 Running 0 20h
2. If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, you might not be able to execute all the following commands.



1. Get the names of the containers that run in the driver pod.
* An IAM API key to allow the SDK to access your account. Create one [here](https://cloud.ibm.com/iam/apikeys).
* Xcode 9.3+
* Swift 4.2+
* iOS 10.0+







 Step 1: Create an Event Notifications service instance 



* Log in to your [IBM Cloud account](https://cloud.ibm.com/).
* In the [IBM Cloud catalog](https://cloud.ibm.com/catalogservices), search Event Notifications > Event Notifications.
* Select a Region from the list of supported regions and select a pricing plan.
* Provide a Service name.
* Select a resource group.
* Accept the licensing agreements and terms by clicking the checkbox.
* Click Create.







 Step 2: Get APNs credentials 

For iOS devices and applications, Apple Push Notification Service (APNs) allows application developers to send remote notifications from Event Notifications service instance on IBM Cloud (the provider) to iOS devices and applications. Messages are sent to a target application on the device.

You need to obtain and configure your APNs credentials. The APNs certificates are securely managed by Event Notifications service and used to connect to APNs server as a provider.



 Registering an App ID 

The App ID (the bundle identifier) is a unique identifier that identifies a specific application. Each application requires an App ID. Services like the Push Notifications service are configured to the App ID.



* Go to the [Apple developer portal](https://developer.apple.com/) and select Certificates, Identifiers & Profiles.

Zoom

![Certificates, Identifiers & Profiles](https://cloud.ibm.com/docs-content/v1/content/4c9a07fe500838e5e55e71e73131ef20b44a6c66/event-notifications/images/en-apns-cert-tab.png)

Figure 2. Certificates, Identifiers & Profiles
* Go to Identifiers > Add identifier button.

Zoom
Add capacity to your cluster such as by [resizing worker pools](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workers) so that the resources can be fulfilled.







 Fixing pod replacement by higher priority pods 

To see if your pod is being replaced by higher priority pods:



1. Get the name of your pod.

oc get pods
2. Describe your pod YAML.

oc get pod <pod_name> -o yaml
3. Check the priorityClassName field.



1. If there is no priorityClassName field value, then your pod has the globalDefault priority class. If your cluster admin did not set a globalDefault priority class, then the default is zero (0), or the lowest priority. Any pod with a higher priority class can preempt, or remove, your pod.
2. If there is a priorityClassName field value, get the priority class.

oc get priorityclass <priority_class_name> -o yaml
3. Note the value field to check your pod's priority.



4. List existing priority classes in the cluster.

oc get priorityclasses
5. For each priority class, get the YAML file and note the value field.

oc get priorityclass <priority_class_name> -o yaml
6. Compare your pod's priority class value with the other priority class values to see if it is higher or lower in priority.
7. Repeat steps 1 to 3 for other pods in the cluster, to check what priority class they are using. If those other pods' priority class is higher than your pod, your pod is not provisioned unless there is enough resources for your pod and every pod with higher priority.
8. Contact your cluster admin to add more capacity to your cluster and confirm that the correct priority classes are assigned.
------------------------------------------------------------
Gradle 6.7.1
------------------------------------------------------------

Build time: 2020-11-16 17:09:24 UTC
Revision: 2972ff02f3210d2ceed2f1ea880f026acfbab5c0

Kotlin: 1.3.72
Groovy: 2.5.12
Ant: Apache Ant(TM) version 1.10.8 compiled on May 10 2020
JVM: 11.0.9 (Eclipse OpenJ9 openj9-0.23.0)
OS: Linux 4.19.121-linuxkit amd64

 oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

 zip
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.

 unzip
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

 git --version
git version 2.17.1

 curl
Why do pods repeatedly fail to restart or are unexpectedly removed? 

Virtual Private Cloud

Classic infrastructure

 What’s happening 

Your pod was healthy but unexpectedly gets removed or gets stuck in a restart loop.

 Why it’s happening 

Your containers might exceed their resource limits, or your pods might be replaced by higher priority pods.

 How to fix it 

See the following sections:



* [Fixing container resource limits](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-failpod-fail-resource-limits)
* [Fixing pod replacement by higher priority pods](https://cloud.ibm.com/docs/containers?topic=containers-ts-app-pod-failpod-fail-higher-priority)





 Fixing container resource limits 



1. Get the name of your pod. If you used a label, you can include it to filter your results.

kubectl get pods --selector='app=wasliberty'
2. Describe the pod and look for the Restart Count.

kubectl describe pod
3. If the pod restarted many times in a short period of time, fetch its status.

kubectl get pod <pod_name> -n <namespace> -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"rnLastState: "}}{{.lastState}}{{end}}'
4. Review the reason. For example, OOM Killed means out of memory indicating that the container is crashing because of a resource limit.
5. Add capacity to your cluster such as by [resizing worker pools](https://cloud.ibm.com/docs/containers?topic=containers-add_workers) so that the resources can be fulfilled.







 Fixing pod replacement by higher priority pods 

To see if your pod is being replaced by higher priority pods:



1. Get the name of your pod.

kubectl get pods
2. Describe your pod YAML.
On macOS, when Skopeo is installed by using the [Homebrew](https://brew.sh/) package manager, the configuration file might be at /opt/homebrew/Cellar/etc/containers/registries.d/default.yaml for Apple silicon, or /usr/local/Cellar/etc/containers/registries.d/default.yaml for Intel.





 Using Podman to sign images 

You can use Podman to sign images. For more information, see [Podman](https://podman.io/).





 Signing images by using the Red Hat OpenShift CLI 

You can sign your images by using the Red Hat OpenShift CLI. For more information, see [Red Hat OpenShift CLI](https://docs.openshift.com/container-platform/3.11/admin_guide/image_signatures.html). The Red Hat® OpenShift® CLI uses the oc command.
If you don't see any helpful information in the events section, check the pod logs for any error messages or other troubleshooting information.

oc logs pod -n <project> <pod>
4. Restart the pod and check if it reaches a Running status.

oc delete pod -n <project> <pod>







 Step 7: Check the system pods 

If the pods are healthy, check if other system pods are experiencing issues. Oftentimes to function properly, one component depends on another component to be healthy.

For example, the OperatorHub has a set of images that are stored in external registries such as quay.io. These images are pulled into the internal registry to use across the projects in your Red Hat OpenShift cluster. If any of the OperatorHub or internal registry components are not set up properly, such as due to lack of permissions or compute resources, the OperatorHub and catalog don't display.



1. Check for pending pods.

oc get pods --all-namespaces | grep Pending
2. Describe the pods and check for the Events.

oc describe pod -n <project_name> <pod_name>

For example, some common messages that you might see from openshift-image-registry pods include:



* A Volume could not be created error message because you created the cluster without the correct storage permission. Red Hat OpenShift on IBM Cloud clusters come with a file storage device by default to store images for the system and other pods. Revise your [infrastructure permissions](https://cloud.ibm.com/docs/openshift?topic=openshift-classic-roles) and restart the pod.
* An order will exceed maximum number of storage volumes allowed error message because you have exceeded the combined quota of file and block storage devices that are allowed per account. [Remove unused storage devices](https://cloud.ibm.com/docs/containers?topic=containers-file_storagestorage_remove_file) or [increase your storage quota](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-managingstoragelimits), and restart the pod.
NAME STATUS ROLES AGE VERSION
10.111.1.11 Ready <none> 1d v1.26+IKS
5. Create a yaml file to deploy a pod onto the node. Make sure to specify the service account you created and the node IP address.

apiVersion: v1
kind: Pod
metadata:
name: testpod
spec:
nodeName: 10.111.1.111
containers:
- image: nginx
name: nginx
serviceAccountName: test-sa

oc apply -f <file_name>
6. Retrieve the service account token from within the pod.



1. Log in to the pod.

kubectl exec testpod -it -- bash
2. Run the following command to get the token. Note that there is no output.

token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)



7. While you are still logged in to the pod, run the command to view the storage metrics. Make sure to specify the node IP address.

curl -k -H "authorization: bearer <token> https://<node_IP>:10250/metrics | grep kubelet_volume_stats
8. View the metrics in the terminal output. You may have to wait several minutes for the metrics to output. If you are still unable to view metrics, [open a support issue](https://cloud.ibm.com/docs/openshift?topic=openshift-get-helphelp-support).
9. After you have finished viewing the metrics, and determined whether the issue is related to dashboard or the metrics agent, delete the configurations and resources that you created in the previous steps.



Do not skip this step.



1. Exit the pod.

exit
2. Delete the pod.

oc delete pod testpod
3. Delete the clusterRoleBinding.
* jmsMdb-3.2
* jndi-1.0
* jpa-2.0
* jpa-2.1
* jpa-2.2
* jsf-2.0
* jsf-2.2
* jsf-2.3
* jsfContainer-2.2
* json-1.0
* jsonb-1.0
* jsonp-1.0
* jsonp-1.1
* jsp-2.2
* jsp-2.3
* jwt-1.0







 L 



* ldapRegistry-3.0
* localConnector-1.0
* logAnalysis-1.0
* logstashCollector-1.0







 M 



* managedBeans-1.0
* mdb-3.1
* mdb-3.2
* mediaServerControl-1.0
* microProfile-1.0
* microProfile-1.2
* microProfile-1.3
* microProfile-1.4
* microProfile-2.0
* microProfile-2.1
* microProfile-2.2
* microProfile-3.0
* microProfile-3.2
* microProfile-3.3
* microProfile-4.0
* microProfile-4.1
* mongodb-2.0
* monitor-1.0







 O 



* oauth-2.0
* openapi-3.0
* openapi-3.1
* openid-2.0
* openidConnectClient-1.0
* openidConnectServer-1.0
* opentracing-1.0
* osgi.jpa-1.0
* osgiAppIntegration-1.0
* osgiBundle-1.0
If the CPU requests are too low, it can possibly fail liveness and readiness probes due to CPU throttling.

Problems with the metrics-server can also be cause problems in other areas. The metrics APIs is not available if the control plane is not able to communicate with the metrics-server by using Konnectivity. Admission control webhooks can prevent the control plane from creating pods, including the metrics-server pod.

 How to fix it 

Follow these steps to troubleshoot.



1. Verify that metrics-server pods exist.

kubectl get pod -n kube-system -l k8s-app=metrics-server

If no pods are listed, there is likely a problem with an admission-control webhook. See [Why do cluster operations fail due to a broken webhook?](https://cloud.ibm.com/docs/containers?topic=containers-webhooks_update).
2. Verify that the apiserver can connect to the metrics-server.

kubectl logs POD -n kube-system -c metrics-server --tail 5

Replace POD with the pod name shown earlier. The content of any logs returned does not matter.

If you get an error message that contains text such as <workerIP>:10250: getsockopt: connection timed out, see [kubectl commands time out](https://cloud.ibm.com/docs/containers?topic=containers-ts_clisexec_logs_fail).
3. If the previous steps do not show a problem, adjust the resources for the metrics-server. See [Adjusting cluster metrics provider resources](https://cloud.ibm.com/docs/containers?topic=containers-kernelmetrics).
For more information, see [Installing Watson Speech to Text](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=text-installing-watson-speech).

Interaction of Speech services with MinIO datastore during installation
: The Speech services runtime components, sttRuntime and ttsRuntime, cannot start until the models and voices for the services are fully uploaded into the MinIO datastore. During installation, the services might fail and automatically restart themselves one or more times until upload of the models and voices is complete. They then start properly. No user action is required.

Defect fix: Correct upgrade documentation
: Defect fix: Documentation for upgrading the Speech services to new versions of IBM Cloud Pak for Data version 4.0.x included incorrect references in some commands. These references are now correct:



* The strings watsonSpeechToTextStatus and watsonTextToSpeechStatus have been changed to speechStatus in both cases.
* The strings status.watsonSpeechToTextVersion and status.watsonTextToSpeechVersion have been changed to .spec.version in both cases.



For more information, see [Upgrading Watson Speech to Text](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=text-upgrading-watson-speech).

Important: Custom language models based on certain next-generation models must be re-created
: Important: If you created custom language models based on certain next-generation models, you must re-create the custom models. Until you re-create the custom language models, speech recognition requests that attempt to use the custom models fail with HTTP error code 400.

You need to re-create custom language models that you created based on the following versions of next-generation models:



* For the en-AU_Telephony model, custom models that you created from en-AU_Telephony.v2021-03-03 to en-AU_Telephony.v2021-10-04.
Move both files to the local executables directory.

$ sudo mv oc /usr/local/bin
$ sudo mv kubectl /usr/local/bin
3. Verify the installation of the IBM Cloud CLI.

ibmcloud dev -v

Example output:

ibmcloud dev version 2.12.0
4. Verify oc.

oc version --client

Example output:

Client Version: 4.8.39
5. Verify kubectl.

kubectl version --client

Example output:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v0.21.0-beta.1", ...







 Install and verify Docker 



1. Install Docker CLI emulator powered by podman.

sudo dnf install podman-docker press 'y' when prompted

Docker daemon is no longer available on Red Hat 8.
2. Verify the Docker CLI emulator.

podman version

Example output:

Client: Podman Engine
Version: 4.0.2
API Version: 4.0.2
Go Version: go1.17.7

Built: Tue Apr 19 05:16:32 2022
OS/Arch: linux/amd64







 Store your login credentials and create a working directory on your jump host 



1. Log in to IBM Cloud for the first time.

Refer to [Configure your environment](https://cloud.ibm.com/docs/cli?topic=cli-getting-startedstep4-configure-idt-env) to learn more about the login command ibmcloud and the --sso parameter.

ibmcloud login [--sso] select the appropriate region, e.g. us-south
2. Create the API key file.