vania-janet/multiturn-rag-retrieval-data huggingface dataset:
When you update worker nodes in [VPC](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_updateportworx_vpc_up) clusters, you must take additional steps to re-attach your Portworx volumes. You can attach your storage volumes by using the API or CLI.



* [Attaching Block Storage for VPC volumes with the CLI](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_storage_att_cr).
* [Attaching Block Storage for VPC volumes with the API](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesvpc_api_attach).



2. To remove a Portworx volume, storage node, or the entire Portworx cluster, see [Portworx cleanup](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_removing).









 Creating a secret to store the KMS credentials 

Before you begin:[Set up encryption](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_encryption)



1. Encode the credentials that you retrieved in the previous section to base64 and note all the base64 encoded values. Repeat this command for each parameter to retrieve the base64 encoded value.

echo -n "<value>" | base64
2. Create a namespace in your cluster called portworx.

kubectl create ns portworx
3. Create a Kubernetes secret named px-ibm in the portworx namespace of your cluster to store your IBM Key Protect information.



1. Create a configuration file for your Kubernetes secret with the following content.

apiVersion: v1
kind: Secret
metadata:
name: px-ibm
namespace: portworx
type: Opaque
data:
IBM_SERVICE_API_KEY: <base64_apikey>
Follow any recovery steps that are in the error message. For example, you might get a message similar to the following, where you must have the correct permissions to the resource group that the cluster is in.

annotations:
workerPoolsConfigStatus: '{"1:3:default":"FAILED CODE: 400
...
"description":"Unable
to validate the request with resource group manager.","type":"Authentication\"recoveryCLI":"To
list available resource groups, run ''ibmcloud resource groups''. Make sure
that your cluster and the other IBM Cloud resources that you are trying to use
are in the same resource group. Verify that you have permissions to work with
the resource group. If you think that the resource group is set up correctly
and you still can't use it, contact IBM Cloud support."}"}'







 Step 3: Review the cluster autoscaler status 

Review the status of the cluster autoscaler.

oc describe cm -n kube-system cluster-autoscaler-status



* status: Review the status message for more troubleshooting information, if any.
* Health: Review the overall health of the cluster autoscaler for any errors or failures.
* ScaleUp: Review the status of scale up activity. In general, if the number of worker nodes that are ready and registered match, the scale up has NoActivity because your worker pool has enough worker nodes.
* ScaleDown: Review the status of scale down activity. If the cluster autoscaler identifies NoCandidates, your worker pool is not scaled down because none of the worker nodes can be removed without taking away requested resources from your workloads.
* Events: Review the events for more troubleshooting information, if any.



Example of a healthy cluster autoscaler status

Data
====
status:
----
Cluster-autoscaler status at 2020-02-04 19:51:50.326683568 +0000 UTC:
Cluster-wide:
Follow any recovery steps that are in the error message. For example, you might get a message similar to the following, where you must have the correct permissions to the resource group that the cluster is in.

annotations:
workerPoolsConfigStatus: '{"1:3:default":"FAILED CODE: 400
...
"description":"Unable
to validate the request with resource group manager.","type":"Authentication\"recoveryCLI":"To
list available resource groups, run ''ibmcloud resource groups''. Make sure
that your cluster and the other IBM Cloud resources that you are trying to use
are in the same resource group. Verify that you have permissions to work with
the resource group. If you think that the resource group is set up correctly
and you still can't use it, contact IBM Cloud support."}"}'







 Step 3: Review the cluster autoscaler status 

Review the status of the cluster autoscaler.

kubectl describe cm -n kube-system cluster-autoscaler-status



* status: Review the status message for more troubleshooting information, if any.
* Health: Review the overall health of the cluster autoscaler for any errors or failures.
* ScaleUp: Review the status of scale up activity. In general, if the number of worker nodes that are ready and registered match, the scale up has NoActivity because your worker pool has enough worker nodes.
* ScaleDown: Review the status of scale down activity. If the cluster autoscaler identifies NoCandidates, your worker pool is not scaled down because none of the worker nodes can be removed without taking away requested resources from your workloads.
* Events: Review the events for more troubleshooting information, if any.



Example of a healthy cluster autoscaler status

Data
====
status:
----
Cluster-autoscaler status at 2020-02-04 19:51:50.326683568 +0000 UTC:
Cluster-wide:
ibmcloud oc worker-pool rebalance --cluster <cluster_name_or_ID> --worker-pool <pool_name>



8. Optional: You can repeat steps 2 - 7 for each worker pool in your cluster. After you complete these steps, all worker nodes in your cluster are set up with the new VLANs.
9. Move networking services to the new VLANs. The networking services in your cluster are still bound to the old VLAN because their IP addresses are from a subnet on that VLAN.



* Routers: Because routers can't be moved across VLANs, you can instead [create router services on the new VLANs and delete router services on the old VLANs](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_routesmigrate-router-vlan-classic).
* Ingress ALBs (Red Hat OpenShift version 3.11 only): Because ALBs can't be moved across VLANs, you can instead [create ALBs on the new VLANs and disable ALBs on the old VLANs](https://cloud.ibm.com/docs/containers?topic=containers-ingress-alb-managemigrate-alb-vlan).



10. Optional: If you no longer need the subnets on the old VLANs, you can [remove them](https://cloud.ibm.com/docs/openshift?topic=openshift-subnetsremove-subnets).
- matchExpressions:
- key: publicVLAN
operator: In
values:
- "2234945"
...
Show more

In the example YAML, the affinity section has publicVLAN as the key and "2234945" as the value.
4. Apply the updated deployment configuration file.

kubectl apply -f with-node-affinity.yaml
5. Verify that the app pods deployed to worker nodes connected to the designated VLAN.



1. List the pods in your cluster. Replace <selector> with the label that you used for the app.

kubectl get pods -o wide app=<selector>

Example output

NAME READY STATUS RESTARTS AGE IP NODE
cf-py-d7b7d94db-vp8pq 1/1 Running 0 10d 172.30.xxx.xxx 10.176.48.78
2. In the output, identify a pod for your app. Note the NODE ID of the worker node that the pod is on.

In the example output of the previous step, the app pod cf-py-d7b7d94db-vp8pq is on worker node 10.176.48.78.
3. List the details for the worker node.

kubectl describe node <worker_node_ID>

Example output

NAME: 10.xxx.xx.xxx
Role:
Labels: arch=amd64
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-south
failure-domain.beta.kubernetes.io/zone=dal10
ibm-cloud.kubernetes.io/encrypted-docker-data=true
kubernetes.io/hostname=10.xxx.xx.xxx
privateVLAN=2234945
publicVLAN=2234967
...
4.
Removing a host node in PIM 

Complete the following steps to remove a host node as a worker in PIM:



1.  Go to Infrastructure > Hardware Nodes.
2.  Select the node that you want to disassociate.
3.  Click Unregister.
4.  Click OK.
You can find your SSH login command in the Apply plan log file in the Jobs section of your workspace. The following command is an example:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -J root@169.63.100.221 root@10.241.4.12
2. The following command and example output shows that the data volume, either /dev/vde or /dev/vdd is mounted to /data on the storage node:

Command:

df -k | grep data

Example output:

/dev/vde       104806400  763756 104042644   1% /data
3. The following command and example output shows that /data is exported as an NFS-shared directory:

Command:

exportfs -v

Example output:

/data          10.248.0.0/16(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
4. In this example, at the NFS client end, the /data directory in the NFS server is mounted to the local directory, /data. The following command and example output shows that the local directory, /data, is mounted to the remote /data directory on the NFS server:

Command:

df -k | grep data

Example output:

/dev/vde       104806400  763756 104042644   1% /data







 Step 7: Add or reduce worker nodes in your HPC cluster 

You can add or reduce the number of worker nodes in your cluster by resizing an existing worker pool, regardless of whether the worker pool is in one zone or spread across multiple zones. For more information, see [Adding worker nodes and zones to clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workers).





 Step 8: Update clusters, worker nodes, and cluster components 

You can install updates to keep your Red Hat OpenShift on IBM Cloud cluster up to date.
When you select a zone that is located outside your country, keep in mind that you might require legal authorization before data can be physically stored in a foreign country.
5. Review the worker node flavors that are available in that zone. The flavor determines the amount of virtual CPU, memory, and disk space that is set up in each worker node and made available to your apps. Worker nodes in classic clusters can be created as virtual machines on shared or dedicated infrastructure, or as bare metal machines that are dedicated to you. For more information, see [Planning your worker node setup](https://cloud.ibm.com/docs/openshift?topic=openshift-planning_worker_nodes). After you create your cluster, you can add different flavors by [adding a worker pool](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersadd_pool).

Before you create a bare metal machine, be sure that you want to provision one. Bare metal machines are billed monthly. If you order a bare metal machine by mistake, you are charged for the entire month, even if you cancel the machine immediately.

ibmcloud oc flavors --zone <zone>
6. Check if you have existing VLANs in the zones that you want to include in your cluster, and note the ID of the VLAN. If you don't have a public or private VLAN in one of the zones that you want to use in your cluster, IBM Cloud Kubernetes Service automatically creates these VLANs for you when you create the cluster.

ibmcloud oc vlan ls --zone <zone>

Example output

ID Name Number Type Router
1519999 vlan 1355 private bcr02a.dal10
1519898 vlan 1357 private bcr02a.dal10
1518787 vlan 1252 public fcr02a.dal10
1518888 vlan 1254 public fcr02a.dal10

If a public and private VLAN already exist, note the matching routers.
The primary node is responsible for queries and log ingestion. The primary node web UI is the single pane of glass for that VMware Aria Operations for Logs Cluster. All queries against data are directed against the primary, which in turn distributes the workload to the workers.
* Worker Node - three nodes minimum are required to form a cluster with the ability to add more workers for scale-out. A worker node ingests logs and stores logs locally.
* Integrated Load Balancer - provides high availability by using proprietary load-balancing configuration (no external load balancer required).
* VMware Aria Operations for Logs Forwarder - deployed to receive logs from the NSX overlay components. Additionally, it can be used by a client if they want to send logs from compute VMs. The VMware Aria Operations for Logs Forwarder is a single VMware Aria Operations for Logs primary node that is used as a remote syslog aggregator to forward alerts to the VMware Aria Operations for Logs cluster. As the VXLAN-backed addresses are outside of the BYOIP address space, NAT rules must be implemented on the NSX ESG.



The following sizes are available and the appropriate one is selected:



* Small – 2,000 events per second
* Medium – 5,000 events per second
* Large – 15,000 events per second



Zoom

![VMware Aria Operations for Logs components diagram](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/vmwaresolutions/images/opsmgmt-vrlicomponents.svg)

Figure 2. VMware Aria Operations for Logs components

VMware Aria Operations for Logs collects logs to provide monitoring information about the environment from a central location.

VMware Aria Operations for Logs collects log events from the following virtual infrastructure and cloud management components (logging clients):



* vCenter
* ESXi hosts
* NSX Manager
* NSX Controllers
* NSX Edge services gateways
* NSX distributed logical router instances
In the previous example output, the app pod cf-py-d7b7d94db-vp8pq is on a worker node with the IP address 10.xxx.xx.xxx.
3. List the worker nodes in the worker pool that you designated in your app deployment.

ibmcloud oc worker ls --cluster <cluster_name_or_ID> --worker-pool <worker_pool_name_or_ID>

Example output

ID Public IP Private IP Machine Type State Status Zone Version
kube-dal10-crb20b637238bb471f8b4b8b881bbb4962-w7 169.xx.xxx.xxx 10.176.48.78 b3c.4x16 normal Ready dal10 1.8.6_1504
kube-dal10-crb20b637238bb471f8b4b8b881bbb4962-w8 169.xx.xxx.xxx 10.176.48.83 b3c.4x16 normal Ready dal10 1.8.6_1504
kube-dal12-crb20b637238bb471f8b4b8b881bbb4962-w9 169.xx.xxx.xxx 10.176.48.69 b3c.4x16 normal Ready dal12 1.8.6_1504

If you created an app affinity rule based on another factor, get that value instead. For example, to verify that the app pod deployed to a worker node on a specific VLAN, view the VLAN that the worker node is on by running ibmcloud oc worker get --cluster <cluster_name_or_ID> --worker <worker_ID>.
4. In the output, verify that the worker node with the private IP address that you identified in the previous step is deployed in this worker pool.









 Deploying an app on a GPU machine
oc get namespaces | grep storage



2. Encode both the ID of your root key and the API key of the service ID that you created to base64.

printf "ROOT-KEY-ID" | base64

printf "SERVICE-ID-API-KEY" | base64
3. Create the Kubernetes secret in the openshift-storage namespace that uses your Hyper Protect Crypto Services credentials.



1. Save the following secret as a YAML file called ibm-hpcs-secret.yaml.

apiVersion: v1
data:
  IBM_KP_CUSTOMER_ROOT_KEY: AaAAAaZAAAAy11AAAyAAkaAaQtAAk0AAA2AzY5AjYaaa67aa your base64 encoded root key ID
  IBM_KP_SERVICE_API_KEY: AAAaaajAAAAAncmAAaaaaAAAAdAAId1AtVjBJRU1aAAaAeTh1aEw=AaaaA your base64 encoded API
kind: Secret
metadata:
  name: ibm-hpcs-secret
  namespace: openshift-storage
type: Opaque
2. Create the secret in your cluster.

oc apply -f ibm-hpcs-secret.yaml



4. Verify that your secret is created.

oc get secrets -A | grep ibm-hpcs-secret







 Preparing your cluster for an OpenShift Data Foundation installation 

Before you install OpenShift Data Foundation, prepare your cluster.

You can't use the OpenShift Data Foundation

[Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).



1. Log in to each worker node in your cluster by using the oc debug command and complete the following steps.



* Log in to the worker node. Replace <worker_node_IP> with the name of your worker node. To get the names of your worker nodes, run the oc get nodes command.
prefix = ""

[registry.mirror]]
location = "us.icr.io/armada-master/rhel8-postgresql-12"
insecure = false
7. Repeat the previous steps to update the registries.conf file on each worker node.







 Step 14: Reboot each worker node 



1. Reboot each worker node in your cluster one at a time.

ibmcloud oc worker reboot -c CLUSTER -w WORKER
2. Wait for each node to reach the Ready status before rebooting the next.







 Step 15: Install the OpenShift Data Foundation add-on from the console 

To install ODF in your cluster, complete the following steps.



1. Before you enable the add-on, review the [change log](https://cloud.ibm.com/docs/openshift?topic=openshift-odf_addon_changelog) for the latest version information. Note that the add-on supports n+1 cluster versions. For example, you can deploy version 4.10.0 of the add-on to an OCP 4.9 or 4.11 cluster. If you have a cluster version other than the default, you must install the add-on from the CLI and specify the --version option.
2. [Review the parameter reference](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy-odf-vpc&interface=uiodf-vpc-param-ref)
3. From the [Red Hat OpenShift clusters console](https://cloud.ibm.com/kubernetes/clusters?platformType=openshift), select the cluster where you want to install the add-on.
4. On the cluster Overview page, find the OpenShift Data Foundation card and click Install. The Install ODF panel opens.
5. In the Install ODF panel, enter the configuration parameters that you want to use for your ODF deployment.
6. Select either Essentials or Advanced as your billing plan.
7.
Back on the cluster Overview tab, follow the same procedure to Connect Monitoring. Use an existing Monitoring instance or create a new instance as shown below:



1. Click Create an instance.
2. Select the same location as to where your cluster is created.
3. Select Graduated Tier as your plan.
4. Enter a unique Service name such as <your-initials>-monitoring.
5. Use the resource group associated with your cluster.
6. Leave IBM platform metrics to Disable and click Create.









 Step 8: Analyze your logs with Log Analysis 

IBM Log Analysis is a cloud native service that you can include as part of your IBM Cloud architecture to add log management capabilities. You can use Log Analysis to manage system and application logs in IBM Cloud. [Learn more](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-getting-started).

This section of the tutorial goes deep into the IBM logging service. You can stop this section at any time and successfully begin the next section.



 Verify that the Log Analysis agent is deployed successfully 

Verify that the Log Analysis-agent pods on each node are in a Running status.

oc get pods -n ibm-observe

The deployment is successful when you see one or more Log Analysis pods:

someone@cloudshell:$ oc get pods -n ibm-observe
NAME READY STATUS RESTARTS AGE
logdna-agent-mdgdz 1/1 Running 0 86s
logdna-agent-qlqwc 1/1 Running 0 86s

The number of Log Analysis pods equals the number of worker nodes in your cluster.



* All pods must be in a Running state
* stdout and stderr are automatically collected and forwarded from all containers. Log data includes application logs and worker logs.
* By default, the Log Analysis agent pod that runs on a worker collects logs from all namespaces on that node.



After the agent is configured logs from this cluster will be visible in the Log Analysis web UI covered in the next section. If after a period of time you cannot see logs, check the agent logs.
You can lock down the public or private interfaces of worker nodes by using Calico network policies for public network or private network isolation. You might need to allow access to the public and private IP addresses of the services that you want to use in these Calico isolation policies.

If your worker nodes need to access services in private networks outside of your IBM Cloud account, you can configure and deploy the strongSwan IPSec VPN service in your cluster or leverage IBM Cloud IBM Cloud Direct Link services to connect to these networks.





 External communication to apps that run on worker nodes with internet-facing workloads 

To expose an app in your cluster to the internet, you can create a public network load balancer (NLB) or Ingress application load balancer (ALB) service. You can improve the security of your cluster by creating a pool of worker nodes that are labeled as edge nodes. The pods for public network services are deployed to the edge nodes so that external traffic workloads are isolated to only a few workers in your cluster. You can further control public traffic to the network services that expose your apps by creating Calico pre-DNAT policies, such as allowlist and blocklist policies.

Ready to get started with a cluster for this scenario? After you plan your [high availability](https://cloud.ibm.com/docs/containers?topic=containers-ha_clusters) and [worker node](https://cloud.ibm.com/docs/containers?topic=containers-planning_worker_nodes) setups, see [Creating clusters](https://cloud.ibm.com/docs/containers?topic=containers-clusters).







 Scenario: Allow limited public connectivity with a gateway appliance 

In this scenario, you want to run workloads in a classic cluster that are accessible to services, databases, or other resources in your on-premises data center. However, you might need to provide limited public access to your cluster, and want to ensure that any public access is controlled and isolated in your cluster. For example, you might need your workers to access an IBM Cloud service that does not support private cloud service endpoints, and must be accessed over the public network. Or you might need to provide limited public access to an app that runs in your cluster.
9. Click Apply.





 Getting your Kubernetes Service subnet IP addresses 

Follow the steps to obtain the correct subnet IP addresses to add to your IAM allowlist.



 Getting your subnet IP addresses in the console 



1. From the IBM Cloud console [resource list](https://cloud.ibm.com/resources), click on your cluster.
2. Click Worker nodes.
3. Note each Public VLAN used by the worker nodes in your cluster. Multiple worker nodes may use the same public VLAN.
4. From the [IBM Cloud console](https://cloud.ibm.com/) menu ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/icons/icon_hamburger.svg), click Classic Infrastructure > IP Management > VLANs.
5. Click on each public VLAN to check if it is used by the worker nodes in your cluster.
6. For each public VLAN that is used by the worker nodes in your cluster, find the Subnets section and note each IP address included in the table. These are the IP addresses that you must include in your allowlist.







 Getting your subnet IP addresses in the CLI 



1. List the public VLANs your worker nodes use. The output is formatted as publicVLAN=<vlan_id>.

oc describe nodes | grep publicVLAN | sort | uniq
2. For each public VLAN, find the associated public subnets. In the output, note the subnet IP addresses in the identifier column.

ibmcloud sl subnet list | grep <vlan-id>

Example output of subnets associated with public VLAN with ID 2761690.

ID identifier type network_space datacenter vlan_id IPs hardware virtual_servers
1962263 169.62.46.56 SECONDARY_ON_VLAN PUBLIC wdc07 2761690 8 0 0