vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You can find the primary IP of your worker node by running ibmcloud ks worker ls.

volumeId and spec.csi.volumeHandle
: In the spec CSI volume attributes section, enter the ID of the Block Storage for Classic volume that you retrieved earlier.

storageClassName
: For the spec storage class name, enter an empty string.

matchExpressions
: In the spec node affinity section, enter the node selector terms to match the zone. For the key, enter failure-domain.beta.kubernetes.io/zone. For the value, enter the zone of your worker node where you want to attach storage.

matchExpressions
: In the spec node affinity section, enter the node selector terms to match the region. For the key, enter failure-domain.beta.kubernetes.io/region. For the value, enter the region of the worker node where you want to attach storage.
6. Create the PV in your cluster.

kubectl apply -f pv.yaml
7. Verify that the PV is created in your cluster.

kubectl get pv
8. Create another configuration file for your PVC. In order for the PVC to match the PV that you created earlier, you must choose the same value for the storage size and access mode. In your storage class field, enter an empty string value to match your PV. If any of these fields don't match the PV, then a new PV and a Block Storage for Classic instance are created automatically via dynamic provisioning.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: <pvc_name>
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <vpc_block_storage_size>
storageClassName: ""
9. Create your PVC.

kubectl apply -f pvc.yaml
10. Verify that your PVC is created and bound to the PV that you created earlier. This process can take a few minutes.
* If you created annotation tasks to track work that you want to do, but none of the annotation work has begun and will not take place until after the workspace is restored, then make a list of the outstanding annotation tasks. Be sure to note the document sets that you imported, but that have not been added to the ground truth yet. Also, make a note of who you assigned to annotate each document set. You will need to reupload these document sets and recreate the annotation tasks after the workspace is restored.



2. Understand tokenizer use.

For machine learning models, workspaces use the machine learning-based tokenizer by default. If you are using a dictionary-based tokenizer and have a specific need to continue doing so, you can configure the workspace to use the dictionary-based tokenizer when you restore it. For more information, see [Tokenizers](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-projectwks_tokenizer).
3. Manage model resources.

Your model, its versions, and snapshot data cannot be migrated. The resources (except read-only dictionaries) that you used to train those artifacts can be migrated. Therefore, after the migration, you can recreate the model. The model that will be produced will perform the same as the models you generated prior to the migration because the resources that are used for training will be the same.

ATTENTION: If you have a model that is already deployed and you plan to delete the workspace after you back it up, withdraw the model from deployment. You can rebuild and redeploy the model after you restore the workspace from the backup.
Change log for worker node fix pack 1.23.16_1561, released 30 January 2023 

The following table shows the changes that are in the worker node fix pack 1.23.16_1561. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.23.15_1558

 Component Previous Current Description 

 Ubuntu 18.04 packages N/A N/A Worker node kernel & package updates for [CVE-2018-20217](https://nvd.nist.gov/vuln/detail/CVE-2018-20217), [CVE-2022-23521](https://nvd.nist.gov/vuln/detail/CVE-2022-23521), [CVE-2022-28321](https://nvd.nist.gov/vuln/detail/CVE-2022-28321), [CVE-2022-40897](https://nvd.nist.gov/vuln/detail/CVE-2022-40897), [CVE-2022-40898](https://nvd.nist.gov/vuln/detail/CVE-2022-40898), [CVE-2022-41903](https://nvd.nist.gov/vuln/detail/CVE-2022-41903), [CVE-2022-42898](https://nvd.nist.gov/vuln/detail/CVE-2022-42898), [CVE-2023-22809](https://nvd.nist.gov/vuln/detail/CVE-2023-22809).
Change log for worker node fix pack 1.18.3_1515, released 8 June 2020 

The following table shows the changes that are in the worker node fix pack 1.18.3_1515. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.18.3_1514

 Component Previous Current Description 

 Ubuntu 18.04 packages N/A N/A Updated worker node images with package updates for [CVE-2019-1547](https://nvd.nist.gov/vuln/detail/CVE-2019-1547), [CVE-2019-1549](https://nvd.nist.gov/vuln/detail/CVE-2019-1549), [CVE-2019-1551](https://nvd.nist.gov/vuln/detail/CVE-2019-1551), [CVE-2019-1563](https://nvd.nist.gov/vuln/detail/CVE-2019-1563), and [CVE-2020-12762](https://nvd.nist.gov/vuln/detail/CVE-2020-12762). 
 Ubuntu 16.04 packages 4.4.0-178-generic 4.4.0-179-generic Updated worker node images with package and kernel updates for [CVE-2019-1547](https://nvd.nist.gov/vuln/detail/CVE-2019-1547), [CVE-2019-1551](https://nvd.nist.gov/vuln/detail/CVE-2019-1551), [CVE-2019-1563](https://nvd.nist.gov/vuln/detail/CVE-2019-1563), and [CVE-2020-12762](https://nvd.nist.gov/vuln/detail/CVE-2020-12762). 







 Change log for 1.18.3_1514, released 26 May 2020
For more information and to migrate existing worker nodes, see [On-demand OCP licenses from Red Hat OpenShift on IBM Cloud](https://cloud.ibm.com/docs/openshift?topic=openshift-costslicenses-on-demand) or review the [IBM Cloud blog announcement](https://www.ibm.com/cloud/blog/announcements/run-workloads-by-the-hour-with-red-hat-openshift-on-ibm-cloud).

Worker node versions
: Worker node fix pack update change log documentation is available.
: Red Hat OpenShift [4.5.17_1519_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4517_1519), [4.4.29_1524_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4429_1524), [4.3.40_1545_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4340_1545), and [3.11.306_1572_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_311311306_1572).





 5 November 2020 

Block Storage for VPC
For more information, see [Gathering cluster data](https://docs.openshift.com/container-platform/4.11/support/gathering-cluster-data.html).

Example command:

oc adm must-gather --image=registry.redhat.io/ocs4/ocs-must-gather-rhel8:latest --dest-dir=ocs_mustgather

You can use the Rook community toolbox to debug issues with your Ceph cluster. For more information, see the [Rook documentation](https://rook.io/docs/rook/v1.3/ceph-toolbox.html).

For more information, review the [common troubleshooting topics](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_openshift_data_foundation).
If you add worker nodes to your cluster and add raw block storage to those workers, restart the Portworx pods on the new worker nodes so that your storage volumes are detected by the DaemonSet.
4. Verify that each storage node is listed with the correct amount of raw block storage by reviewing the Capacity column in the Cluster Summary section of your CLI output.
5. Review the Portworx I/O classification that was assigned to the disks that are part of the Portworx cluster. During the setup of your Portworx cluster, every disk is inspected to determine the performance profile of the device. The profile classification depends on how fast the network is that your worker node is connected to and the type of storage device that you have. Disks of SDS worker nodes are classified as high. If you manually attach disks to a virtual worker node, then these disks are classified as low due to the slower network speed that comes with virtual worker nodes.

kubectl exec -it <portworx_pod> -n kube-system -- /opt/pwx/bin/pxctl cluster provision-status

Example output

NODE NODE STATUS POOL POOL STATUS IO_PRIORITY SIZE AVAILABLE USED PROVISIONED RESERVEFACTOR ZONE REGION RACK
10.184.58.11 Up 0 Online LOW 20 GiB 17 GiB 3.0 GiB 0 B 0 dal12 us-south default
10.176.48.67 Up 0 Online LOW 20 GiB 17 GiB 3.0 GiB 0 B 0 dal10 us-south default
10.176.48.83 Up 0 Online HIGH 3.5 TiB 3.5 TiB 10 GiB 0 B 0 dal10 us-south default







 Creating a Portworx volume 

Start creating Portworx volumes by using [Kubernetes dynamic provisioning](https://cloud.ibm.com/docs/containers?topic=containers-kube_conceptsdynamic_provisioning).



1. List available storage classes in your cluster and check whether you can use an existing Portworx storage class that was set up during the Portworx installation. The pre-defined storage classes are optimized for database usage and to share data across pods.
Change log for worker node fix pack 1.17.6_1527, released 8 June 2020 

The following table shows the changes that are in the worker node fix pack 1.17.6_1527. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.17.6_1526

 Component Previous Current Description 

 Ubuntu 18.04 packages N/A N/A Updated worker node images with package updates for [CVE-2019-1547](https://nvd.nist.gov/vuln/detail/CVE-2019-1547), [CVE-2019-1549](https://nvd.nist.gov/vuln/detail/CVE-2019-1549), [CVE-2019-1551](https://nvd.nist.gov/vuln/detail/CVE-2019-1551), [CVE-2019-1563](https://nvd.nist.gov/vuln/detail/CVE-2019-1563), and [CVE-2020-12762](https://nvd.nist.gov/vuln/detail/CVE-2020-12762). 
 Ubuntu 16.04 packages 4.4.0-178-generic 4.4.0-179-generic Updated worker node images with package and kernel updates for [CVE-2019-1547](https://nvd.nist.gov/vuln/detail/CVE-2019-1547), [CVE-2019-1551](https://nvd.nist.gov/vuln/detail/CVE-2019-1551), [CVE-2019-1563](https://nvd.nist.gov/vuln/detail/CVE-2019-1563), and [CVE-2020-12762](https://nvd.nist.gov/vuln/detail/CVE-2020-12762). 







 Change log for 1.17.6_1526, released 26 May 2020
Change log for worker node fix pack 1.15.11_1538, released 11 May 2020 

The following table shows the changes that are in the worker node fix pack 1.15.11_1538. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.15.11_1537

 Component Previous Current Description 

 Ubuntu 18.04 packages 4.15.0-96-generic 4.15.0-99-generic Updated worker node images with kernel and package updates for [CVE-2019-19768](https://nvd.nist.gov/vuln/detail/CVE-2019-19768), [CVE-2020-11884](https://nvd.nist.gov/vuln/detail/CVE-2020-11884), and [CVE-2020-12243](https://nvd.nist.gov/vuln/detail/CVE-2020-12243). 
 Ubuntu 16.04 packages 4.4.0-177-generic 4.4.0-178-generic Updated worker node images with package and kernel updates for [CVE-2019-19768](https://nvd.nist.gov/vuln/detail/CVE-2019-19768) and [CVE-2020-12243](https://nvd.nist.gov/vuln/detail/CVE-2020-12243). 







 Change log for worker node fix pack 1.15.11_1537, released 27 April 2020 

The following table shows the changes that are in the worker node fix pack 1.15.11_1537. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.15.11_1535

 Component Previous Current Description
You can list the existing custom labels on worker nodes in the worker pool by running ibmcloud ks worker-pool get -c <cluster_name_or_ID> --worker-pool <pool>.

-p, --worker-pool WORKER_POOL
: Required: The name of the worker node pool that you want to view the details of. To list available worker pools, run ibmcloud ks worker-pool ls --cluster <cluster_name_or_ID>.

-f
: Optional: Force the command to run with no user prompts.

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker-pool label set command 

ibmcloud ks worker-pool label set --worker-pool pool1 --cluster my_cluster --label app=dev







 ibmcloud ks worker-pool ls 

Virtual Private Cloud

Classic infrastructure

List all worker pools in a cluster.

ibmcloud ks worker-pool ls --cluster CLUSTER [--output json] [-q]

Minimum required permissions
: Viewer platform access role for the cluster in IBM Cloud Kubernetes Service

Command options:

-c, --cluster CLUSTER_NAME_OR_ID
: Required: The name or ID of the cluster for which you want to list worker pools.

--output json
: Optional: Prints the command output in JSON format.

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker-pool ls command 

ibmcloud ks worker-pool ls --cluster my_cluster







 ibmcloud ks worker-pool rebalance 

Virtual Private Cloud

Classic infrastructure

Rebalance a worker pool in a cluster after you delete a worker node. When you run this command, a new worker or workers are added to your worker pool so that the worker pool has the same number of nodes per zone that you specified.

ibmcloud ks worker-pool rebalance --cluster CLUSTER --worker-pool WORKER_POOL [-q]

Minimum required permissions
: Operator platform access role for the cluster in IBM Cloud Kubernetes Service

Command options:
prod-dal10-pa215dcf5bbc0844a990fa6b0fcdbff286-w2 192.0.2.27 10.100.10.15 u3c.2x4 normal Ready
2. If a random NodePort was assigned, find out which one was assigned.

oc describe service <service_name>

Output

NAME: <service_name>
Namespace: default
Labels: run=<deployment_name>
Selector: run=<deployment_name>
Type: NodePort
IP: 10.10.10.8
Port: <unset> 8080/TCP
NodePort: <unset> 30872/TCP
Endpoints: 172.30.171.87:8080
Session Affinity: None
No events.

In this example, the NodePort is 30872.

If the Endpoints section displays <none>, check the <selectorkey> and <selectorvalue> that you use in the spec.selector section of the NodePort service. Ensure that it is the same as the key/value pair that you used in the spec.template.metadata.labels section of your deployment YAML.
3. Form the URL with one of the worker node IP addresses and the NodePort. Example: http://192.0.2.23:30872. For VPC clusters, you must be connected to the private network, such as through a VPN connection, to access the worker node private IP address and NodePort.
GET/v2/vpc/getWorkerPool View details of a worker pool for a VPC cluster. containers-kubernetes.cluster.read N/A 
 GET/v2/vpc/getWorkerPools View details of a worker pool for a VPC cluster. containers-kubernetes.cluster.read N/A 
 GET/v2/vpc/getWorkers View all workers for VPC cluster. containers-kubernetes.cluster.read N/A 
 PATCH/v1/clusters/{idOrName}/workerpools/{poolidOrName} Resize or rebalance a worker pool. containers-kubernetes.cluster.operate containers-kubernetes.workerpool.update 
 PATCH/v1/clusters/{idOrName}/workerpools/{poolidOrName}/zones/{zoneid} Updates network configuration for a worker pool for a given zone. containers-kubernetes.cluster.operate containers-kubernetes.zone.update 
 POST/v1/clusters/{idOrName}/workerpools Create a worker pool for a cluster. containers-kubernetes.cluster.operate containers-kubernetes.workerpool.create 
 POST/v1/clusters/{idOrName}/workerpools/{poolidOrName}/zones Add a zone to the specified worker pool for a cluster. containers-kubernetes.cluster.operate containers-kubernetes.workerpool.create 
 POST/v1/clusters/{idOrName}/workers Add worker nodes to a cluster. containers-kubernetes.cluster.operate containers-kubernetes.worker.create 
 POST/v2/rebalanceWorkerPool Rebalance workers in a worker pool. containers-kubernetes.cluster.operate containers-kubernetes.account.update 
 POST/v2/removeWorker Delete a worker node from a cluster. containers-kubernetes.cluster.operate containers-kubernetes.account.delete 
 POST/v2/removeWorkerPool Removes a worker pool.
: If you want to use existing storage that you provisioned earlier, but never used in your cluster before, you must make the storage available in the same subnet as your worker nodes.



1. From the [IBM Cloud infrastructure portal](https://cloud.ibm.com/classic), click Storage.
2. Click File Storage for Classic and from the Actions menu, select Authorize Host.
3. Select Subnets.
4. From the drop-down list, select the private VLAN subnet that your worker node is connected to. To find the subnet of your worker node, run ibmcloud ks worker ls --cluster <cluster_name> and compare the Private IP of your worker node with the subnet that you found in the drop-down list.
5. Click Submit.
6. Click the name of the File Storage for Classic.
7. Note the Mount Point, the size, and the Location field. The Mount Point field is displayed as <nfs_server>:<file_storage_path>.







 Creating a persistent volume and a persistent volume claim 



1. Create a storage configuration file for your PV. Include the values that you retrieved earlier.

apiVersion: v1
kind: PersistentVolume
metadata:
name: mypv
labels:
failure-domain.beta.kubernetes.io/region: <region>
failure-domain.beta.kubernetes.io/zone: <zone>
spec:
capacity:
storage: "<size>"
accessModes:
- ReadWriteMany
nfs:
server: "<nfs_server>"
path: "<file_storage_path>"

name
: Enter the name of the PV object to create.

labels
: Enter the region and the zone that you retrieved earlier. You must have at least one worker node in the same region and zone.

storage
: Enter the storage size of the existing NFS file share that you retrieved earlier.
To update the version of the operating system that a worker node uses, such as from Ubuntu 16 to 18, you can [replace the flavor of the worker pool](https://cloud.ibm.com/docs/containers?topic=containers-updatemachine_type).

You can also log in to your cluster to check the operating system of the worker nodes.



1. [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)
2. List your worker nodes.

kubectl get nodes
3. Describe your worker node and check for the operating system label that IBM applies, or the OS Image and Operating System fields in the System Info section.

kubectl describe node <node>

Example output

NAME: 10.xxx.xx.xxx
Roles: <none>
Labels: arch=amd64
...
ibm-cloud.kubernetes.io/os=UBUNTU_18_64
...
kubernetes.io/arch=amd64
kubernetes.io/hostname=10.189.33.198
kubernetes.io/os=linux
...
System Info:
OS Image: Ubuntu 18.04.5 LTS
Operating System: linux
Architecture: amd64
...









 Virtual machines 

With VMs, you get greater flexibility, quicker provisioning times, and more automatic scalability features than bare metal, at a more cost-effective price. You can use VMs for most general-purpose use cases such as testing and development environments, staging, and prod environments, microservices, and business apps. However, there is a tradeoff in performance. If you need high-performance computing for data- or RAM-intensive workloads, consider creating classic clusters with [bare metal](https://cloud.ibm.com/docs/containers?topic=containers-planning_worker_nodesbm) worker nodes.
kubectl describe node <node>
2. Cordon the node that isn't Ready so that you can start investigating.
3. Drain the worker node. Review the [Kubernetes documentation to safely drain pods from your worker node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/).

kubectl drain <node>
4. [Reload](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reload) or [reboot](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reboot) your worker node.











 Zone alerts 

To set up zone level alerts, edit the sysdig-agent ConfigMap to include the required label filters.



1. Edit the ConfigMap by running the following command.

kubectl edit configmap sysdig-agent -n ibm-observe
2. Add the following YAML block after k8s_cluster_name: <cluster_name>. Replace <cluster_name> with the name of the cluster that you want to you want to monitor.

k8s_labels_filter:
- include: "kubernetes.node.label.kubernetes.io/hostname"
- include: "kubernetes.node.label.kubernetes.io/role"
- include: "kubernetes.node.label.ibm-cloud.kubernetes.io/zone"
- exclude: ".kubernetes.io/"
- exclude: ".pod-template-hash"
- exclude: ".pod-template-generation"
- exclude: ".controller-revision-hash"
- include: ""
3. Restart the IBM Cloud Monitoring pods. Delete all the pods and wait for them to restart. Get the list of pods.

kubectl get pods -n ibm-observe
Load balancer and load balancer monitor for IBM Cloud Provider 1916 1998 Updated Go to version 1.17.10 and updated dependencies. 







 Change log for worker node fix pack 1.23.6_1530, released 23 May 2022 

The following table shows the changes that are in the worker node fix pack 1.23.6_1530. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.23.6_1529

 Component Previous Current Description
Load balancer and load balancer monitor for IBM Cloud Provider 2383 2420 Updated the image to resolve CVEs. 







 Change log for worker node fix pack 1.23.17_1570, released 27 March 2023 

The following table shows the changes that are in the worker node fix pack 1.23.17_1570. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.23.16_1568

 Component Previous Current Description 

 Ubuntu 18.04 packages N/A N/A Worker node package updates for [CVE-2021-36222](https://nvd.nist.gov/vuln/detail/CVE-2021-36222), [CVE-2021-37750](https://nvd.nist.gov/vuln/detail/CVE-2021-37750), [CVE-2022-47024](https://nvd.nist.gov/vuln/detail/CVE-2022-47024), [CVE-2023-0049](https://nvd.nist.gov/vuln/detail/CVE-2023-0049), [CVE-2023-0054](https://nvd.nist.gov/vuln/detail/CVE-2023-0054), [CVE-2023-0288](https://nvd.nist.gov/vuln/detail/CVE-2023-0288), [CVE-2023-0433](
Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.17.3_1516

 Component Previous Current Description 

 GPU worker node provisioning N/A N/A Fixed bug where new GPU worker nodes could not automatically join clusters that run Kubernetes version 1.16 or 1.17. 
 Ubuntu 18.04 packages 4.15.0-76-generic 4.15.0-88-generic Updated worker node images with package updates for [CVE-2019-5108](https://nvd.nist.gov/vuln/detail/CVE-2019-5108), [CVE-2019-20096](https://nvd.nist.gov/vuln/detail/CVE-2019-20096), [CVE-2019-18885](https://nvd.nist.gov/vuln/detail/CVE-2019-18885), [CVE-2019-19082](https://nvd.nist.gov/vuln/detail/CVE-2019-19082), [CVE-2019-19078](https://nvd.nist.gov/vuln/detail/CVE-2019-19078), [CVE-2019-19332](https://nvd.nist.gov/vuln/detail/CVE-2019-19332), [CVE-2016-9840](https://nvd.nist.gov/vuln/detail/CVE-2016-9840), [CVE-2016-9841](https://nvd.nist.gov/vuln/detail/CVE-2016-9841), [CVE-2016-9842](https://nvd.nist.gov/vuln/detail/CVE-2016-9842), and [CVE-2016-9843](https://nvd.nist.gov/vuln/detail/CVE-2016-9843).