vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you choose to manually assign hosts, you must [label your worker nodes](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driver&interface=uiazure-disk-label-nodes) before creating your storage configuration.
2. [Set up Satellite Config](https://cloud.ibm.com/docs/satellite?topic=satellite-setup-clusters-satconfig).
3. [Create a Satellite cluster](https://cloud.ibm.com/docs/satellite?topic=openshift-satellite-clusters) that runs on compute hosts in Azure. Make sure that you select the Enable cluster admin access for Satellite Config option when you create the cluster. If you don't enable Administrator (admin) access for Satellite Config when creating your cluster, you must re-create your cluster and enable admin access before you can deploy storage.
4. [Create your configuration file](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driver&interface=uiazuredisk-csi-driver-config-create-console).





 Optional: Labeling your worker nodes for Azure Disk 

Complete the following steps to add the required labels to your worker nodes for the Azure Disk CSI driver template.

If you manually assigned your Azure hosts to your Location and did not use the Schematics template in the console, you must [label your worker nodes](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driver&interface=uiazure-disk-label-nodes) before creating your storage configuration.



1. List your Azure worker nodes and make a note of the name of each node.

oc get nodes
2. Get the details of each node and make a note of the zone that the node is in. For example: eastus-1.

oc get nodes <node-name> -o yaml | grep zone
3. Label your worker nodes with the zone value that you retrieved earlier.
Removing Satellite worker nodes or clusters 

When you remove Red Hat OpenShift clusters or worker nodes in a cluster, the hosts that provide the compute capacity for your worker nodes are not automatically deleted. Instead, the hosts remain attached to your Satellite location, but require you to reload the host to be able to reassign the hosts to other Satellite resources.



1. Back up any data that runs in the worker node or cluster that you want to save. For example, you might save a copy of all the data in your cluster and upload these files to a persistent storage solution, such as IBM Cloud Object Storage.

oc get all --all-namespaces -o yaml
2. Get the Worker ID of each host in your cluster.

ibmcloud sat host ls --location <satellite_location_name_or_ID>

Example output

Retrieving hosts...
OK
Name ID State Status Cluster Worker ID Worker IP
machine-name-1 aaaaa1a11aaaaaa111aa assigned Ready infrastructure sat-virtualser-4d7fa07cd3446b1f9d8131420f7011e60d372ca2 169.xx.xxx.xxx
machine-name-2 bbbbbbb22bb2bbb222b2 assigned Ready infrastructure sat-virtualser-9826f0927254b12b4018a95327bd0b45d0513f59 169.xx.xxx.xxx
machine-name-3 ccccc3c33ccccc3333cc assigned Ready mycluster12345 sat-virtualser-948b454ea091bd9aeb8f0542c2e8c19b82c5bf7a 169.xx.xxx.xxx
3. Remove the worker nodes or clusters by referring to the following options. The corresponding hosts in your Satellite location become unassigned and require a reload before you can use them for other Satellite resources.
[In the IBM Cloud console](https://cloud.ibm.com), select Observability from the menu.
2. Select Monitoring.
3. In the row of the instance for Block Storage for VPC alerts, select Open dashboard.
4. From the menu, select Get started.
5. Under the Install the Agent section, select Add Sources.
6. Follow the instructions to install the agent.
7. Make sure the agent is running by using the kubectl get pods -n CLUSTER_NAME | grep syslog command.



3. Configure the notification channels.



1. In the IBM Cloud Monitoring dashboard, select Monitoring Operations > Settings.
2. Select Notification Channels > Add Notification Channel and pick one of the available notification methods.
3. Complete the settings for the new channel and click Save.
4. Optional: Repeat the previous steps to add more channels.



4. Create an alert.



1. In the IBM Cloud Monitoring dashboard, select Alerts > Library.
2. Choose one of the templates and select Enable Alert. For example, you can search for PVC storage and enable the PVC Storage Usage Is Reaching The Limit alert.
3. Customize the alert settings on the template and select Enable Alert to apply your settings.





If your Block Storage for VPC volumes are reaching capacity, you can [set up volume expansion](https://cloud.ibm.com/docs/containers?topic=containers-vpc-blockvpc-block-volume-expand).







 Monitoring worker node health in with Autorecovery 

The Autorecovery system uses various checks to query worker node health status. If Autorecovery detects an unhealthy worker node based on the configured checks, Autorecovery triggers a corrective action like rebooting a VPC worker node or reloading the operating system in a classic worker node. Only one worker node undergoes a corrective action at a time. The worker node must complete the corrective action before any other worker node undergoes a corrective action.
helm install ibm-object-storage-plugin ./ibm-object-storage-plugin --set dcname="${DC_NAME}" --set provider="${CLUSTER_PROVIDER}" --set workerOS="${WORKER_OS}" --region="${REGION} --set platform="${PLATFORM}" --set license=true [--set bucketAccessPolicy=false]

DC_NAME
: The cluster data center. To retrieve the data center, run kubectl get cm cluster-info -n kube-system -o jsonpath="{.data.cluster-config.json}{'n'}". Store the data center value in an environment variable by running SET DC_NAME=<datacenter>. Optional: Set the environment variable in Windows PowerShell by running $env:DC_NAME="<datacenter>".

CLUSTER_PROVIDER
: The infrastructure provider. To retrieve this value, run kubectl get nodes -o jsonpath="{.items[].metadata.labels.ibm-cloud.kubernetes.io/iaas-provider}{'n'}". If the output from the previous step contains softlayer, then set the CLUSTER_PROVIDER to "IBMC". If the output contains gc, ng, or g2, then set the CLUSTER_PROVIDER to "IBMC-VPC". Store the infrastructure provider in an environment variable. For example: SET CLUSTER_PROVIDER="IBMC-VPC".

WORKER_OS and PLATFORM
: The operating system of the worker nodes. To retrieve these values, run kubectl get nodes -o jsonpath="{.items[].metadata.labels.ibm-cloud.kubernetes.io/os}{'n'}". Store the operating system of the worker nodes in an environment variable. For IBM Cloud Kubernetes Service clusters, run SET WORKER_OS="debian" and SET PLATFORM="k8s".

REGION
: The region of the worker nodes.
Worker node flavors with vGPU support are the gx2 flavor class, for example: gx2.16x128.2v100. IBM Cloud VPC worker nodes with GPUs are available for allowlisted accounts only. To request that your account be allowlisted, see [Requesting access to allowlisted features](https://cloud.ibm.com/docs/openshift?topic=openshift-get-help). Be sure to include the data centers, the VPC infrastructure profile, and the number of workers that you want use. For example 12 worker nodes in us-east-1 of VPC profile gx2-16x128xv100.





 12 September 2022 

Block Storage for VPC add-on version 4.3.6_1579 is available.
: For more information, see [version 4.3.6_1579](https://cloud.ibm.com/docs/containers?topic=containers-vpc_bs_changelog)

IBM Cloud Object Storage plug-in version 2.2.0 is available.
: For more information, see the [change log](https://cloud.ibm.com/docs/containers?topic=containers-cos_plugin_changelog).

CLI change log update
: The [CLI change log](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_changelog) is updated for version 1.0.446.

ALB OAuth Proxy add-on
: Version 2.0.0_1214 of the [ALB OAuth Proxy add-on](https://cloud.ibm.com/docs/containers?topic=containers-alb-oauth-proxy-changelog) is released.





 8 September 2022 

CLI change log update
: The [CLI change log](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_changelog) is updated for version 1.0.444.





 7 September 2022 

Master fix pack update
When you finish troubleshooting, be sure to follow the steps in the [Cleaning up after debugging](https://cloud.ibm.com/docs/containers?topic=containers-cs_ssh_workerssh-debug-cleanup) section to disable SSH access.



1. Choose an existing or create a new public SSH key.

ssh-keygen -t rsa -b 4096
ls /.ssh

id_rsa id_rsa.pub

cat /.ssh/id_rsa.pub
2. Get the name of the worker node that you want to access. The worker node name is its private IP address.

kubectl get nodes -o wide
3. Create the following YAML file for a debug pod, and save the file as enable-ssh.yaml. Replace <NODE_NAME> with the worker node name and replace the example value for SSH_PUBLIC_KEY with your public SSH key. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

apiVersion: v1
kind: Pod
metadata:
name: enable-ssh-<NODE_NAME>
namespace: default
spec:
tolerations:
- operator: "Exists"
hostNetwork: true
hostPID: true
hostIPC: true
containers:
- image: docker.io/library/alpine:latest
env:
- name: SSH_PUBLIC_KEY
value: "<ssh-rsa AAA...ZZZ user@ibm.com>"
If you choose to manually assign hosts, you must [label your worker nodes](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driverazure-disk-label-nodes) before creating your storage configuration.
2. [Set up Satellite Config](https://cloud.ibm.com/docs/satellite?topic=satellite-setup-clusters-satconfig).
3. [Create a Satellite cluster](https://cloud.ibm.com/docs/satellite?topic=openshift-satellite-clusters) that runs on compute hosts in Azure. Make sure that you select the Enable cluster admin access for Satellite Config option when you create the cluster. If you don't enable Administrator (admin) access for Satellite Config when creating your cluster, you must re-create your cluster and enable admin access before you can deploy storage.
4. [Create your configuration file](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driverazuredisk-csi-driver-config-create-console).





 Optional: Labeling your worker nodes for Azure Disk 

Complete the following steps to add the required labels to your worker nodes for the Azure Disk CSI driver template.

If you manually assigned your Azure hosts to your Location and did not use the Schematics template in the console, you must [label your worker nodes](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driverazure-disk-label-nodes) before creating your storage configuration.



1. List your Azure worker nodes and make a note of the name of each node.

oc get nodes
2. Get the details of each node and make a note of the zone that the node is in. For example: eastus-1.

oc get nodes <node-name> -o yaml | grep zone
3. Label your worker nodes with the zone value that you retrieved earlier. Replace <node-name> and <zone> with the node name and zone of your worker node.
* Replace <logging_public_IP> with the [Log Analysis IP addresses](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-endpointsendpoints_api_public).









 Next steps 

If you use load balancer services, ensure that all traffic that uses the VRRP protocol is allowed between worker nodes on the public and private interfaces. Red Hat OpenShift on IBM Cloud uses the VRRP protocol to manage IP addresses for public and private load balancers.

If you use Ingress or routes to expose apps in your cluster, allow incoming network traffic from [Akamai's source IP addresses](https://github.com/IBM-Cloud/kube-samples/tree/master/akamai/gtm-liveness-test) on port 80 to the IP addresses of your router services so that the Red Hat OpenShift control plane can check the health of your routers.







 Opening required ports in a private allowlist 

If you have an allowlist on the private network in your IBM Cloud infrastructure account, such as a Virtual Router Appliance (Vyatta), you must open IP ranges, ports, and protocols in your allowlist to allow worker nodes to communicate with the master, with each other, with infrastructure resources, and with other IBM Cloud services.

Before you begin



1. Allow the IBM Cloud infrastructure private IP ranges so that you can create worker nodes in your cluster.



1. Allow the appropriate IBM Cloud infrastructure private IP ranges. See [Backend (private) Network](https://cloud.ibm.com/docs/cloud-infrastructure?topic=cloud-infrastructure-ibm-cloud-ip-rangesback-end-network).
2. Allow the IBM Cloud infrastructure private IP ranges for all the [zones](https://cloud.ibm.com/docs/openshift?topic=openshift-regions-and-zoneslocations) that you are using.
From the cluster Worker Nodes tab, find the worker node with an IP address that matches the IP address of the host that you want to remove.
4. Select the worker node.
5. From the table action menu, click Delete.
6. In the confirmation message, clear the option to replace the worker node and click Delete.
7. Return to the Satellite Locations > Hosts table.



5. From the Hosts table, hover over the host that you want to remove and click the Action menu icon ![Action menu icon](https://cloud.ibm.com/docs-content/v1/content/c8335d66af691d19e837116fd633481c477061eb/icons/action-menu-icon.svg).
6. Click Remove host, enter the host name to confirm deletion, and click Remove.
7. Follow the instructions from your underlying infrastructure provider to complete one of the following actions:



* To reuse the host for other purposes, reload the operating system of the host. For example, you might reattach the host to a Satellite location later. When you reattach a host, the host name can remain the same as the previous name, but a new host ID is generated.
* To no longer use the host, delete the host from your infrastructure provider.





After the host is removed, shut it down and do not activate it again without reloading the OS.





 Removing hosts with the CLI 

Use the CLI plug-in for Satellite commands to remove your hosts as compute capacity from the Satellite location.

Removing a host cannot be undone. Before you remove a host, make sure that your cluster or location control plane has enough compute resources to continue running even after you remove the host, or back up any data that you want to keep. Note that the underlying host infrastructure is not deleted.



1. Make sure that your cluster or location control plane has enough compute resources to continue running even after you remove the host, or back up any data that you want to keep.
2. Log in your IBM Cloud account.
: Worker node patch updates are available for Kubernetes [1.15.3_1516](https://cloud.ibm.com/docs/containers?topic=containers-115_changelog1153_1516_worker), [1.14.6_1532](https://cloud.ibm.com/docs/containers?topic=containers-114_changelog1146_1532_worker), [1.13.10_1535](https://cloud.ibm.com/docs/containers?topic=containers-113_changelog11310_1535_worker), [1.12.10_1566](https://cloud.ibm.com/docs/containers?topic=containers-112_changelog11210_1566_worker), and Red Hat OpenShift [3.11.135_1523](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_311311135_1523_worker).







 August 2019 



 29 August 2019 

Forwarding Kubernetes API audit logs to IBM Log Analysis
: Added steps to [create an audit webhook in your cluster](https://cloud.ibm.com/docs/containers?topic=containers-health-audit) to collect Kubernetes API audit logs from your cluster and forward them to IBM Log Analysis.





 28 August 2019 

CLI change log
: Updated the IBM Cloud Kubernetes Service CLI plug-in change log page for the [release of version 0.3.112](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_changelog).

Version change logs
Generally, assign at least 3 hosts as worker nodes in your cluster.



6. Verify that your cluster reaches a normal state.

ibmcloud oc cluster ls

Example output

OK
Name ID State Created Workers Location Version Resource Group Name Provider
satcluster brkhsd220b6ktv7sjl50 normal 2 hours ago 3 mylocation 4.3.23_1525_openshift Default satellite
7. [Access your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_clusteraccess_cluster_sat) to run oc and kubectl commands or access the Red Hat OpenShift web console. If you enabled Satellite Config access, you must complete this step to synchronize the permissions.

If your location hosts have private network connectivity only, or if you use Amazon Web Services, Google Cloud Platform, or Microsoft Azure hosts, you must be connected to your hosts' private network, such as through VPN access, to connect to your cluster and access the Red Hat OpenShift web console. Alternatively, if your hosts have public network connectivity, you can test access to your cluster by changing your cluster's and location's DNS records to [use your hosts' public IP addresses](https://cloud.ibm.com/docs/openshift?topic=openshift-access_clustersat_public_access).
8. Optional: [Set up the internal container image registry](https://cloud.ibm.com/docs/openshift?topic=openshift-satellite-clusterssatcluster-internal-registry).



After you [access your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_clusteraccess_cluster_sat) and run oc get nodes or oc describe node <worker_node>, you might see that the worker nodes have master,worker roles. In OpenShift Container Platform clusters, operators use the master role as a nodeSelector so that OCP can deploy default components that are controlled by operators, such as the internal registry, in your cluster.
: Worker node fix pack update change log documentation is available for Kubernetes version [1.19.4_1529](https://cloud.ibm.com/docs/containers?topic=containers-changelog_1191194_1529), [1.18.12_1535](https://cloud.ibm.com/docs/containers?topic=containers-118_changelog11812_1535), [1.17.14_1548](https://cloud.ibm.com/docs/containers?topic=containers-117_changelog11714_1548), and [1.16.15_1554](https://cloud.ibm.com/docs/containers?topic=containers-116_changelog11615_1554).





 9 December 2020 

Accessing clusters
: Updated the steps for [accessing clusters through the private cloud service endpoint](https://cloud.ibm.com/docs/containers?topic=containers-access_clusteraccess_private_se) to use the --endpoint private option in the ibmcloud ks cluster config command.

CLI change log
: Updated the IBM Cloud Kubernetes Service CLI plug-in change log page for the [release of version 1.0.206](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_changelog10).

Proxy protocol for Ingress
: In VPC clusters, you can now [enable the PROXY protocol](https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotationspreserve_source_ip_vpc) for all load balancers that expose Ingress ALBs in your cluster. The PROXY protocol enables load balancers to pass client connection information that is contained in headers on the client request, including the client IP address, the proxy server IP address, and both port numbers, to ALBs.

Helm version 2 unsupported
: Removed all steps for using Helm v2, which is unsupported.
* Example command for Bare Metal worker nodes: ip link set dev bond0 mtu 9000;ip link set dev bond1 mtu 9000;
* Example command VPC Gen 2 worker nodes: ip link set dev ens3 mtu 9000;



apiVersion: apps/v1
kind: DaemonSet
metadata:
name: jumbo-apply
namespace: kube-system
labels:
tier: management
app: jumbo-apply
spec:
selector:
matchLabels:
name: jumbo-apply
template:
metadata:
labels:
name: jumbo-apply
spec:
hostNetwork: true
hostPID: true
hostIPC: true
tolerations:
- operator: Exists
initContainers:
- command:
- sh
- -c
- ip link set dev bond0 mtu 9000;ip link set dev bond1 mtu 9000; Update this command based on your worker node type.
image: alpine:3.6
imagePullPolicy: IfNotPresent
name: iplink
resources: {}
securityContext:
privileged: true
capabilities:
add:
- NET_ADMIN
volumeMounts:
- name: modifysys
mountPath: /sys
containers:
- resources:
requests:
cpu: 0.01
image: alpine:3.6
name: sleepforever
command: ["/bin/sh", "-c"]
args:
- >
while true; do
sleep 100000;
done
volumes:
- name: modifysys
hostPath:
path: /sys
Show more



 Updating the Calico ConfigMap 

After [applying the DaemonSet to increase the Calico plug-in MTU](https://cloud.ibm.com/docs/containers?topic=containers-kernelcalico-mtu), complete the following steps to update the Calico ConfigMap.



1.
At least one worker node in the cluster is not available, but other worker nodes are available and can take over the workload. Try to [reload](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reload) the unavailable worker nodes. - Your cluster has zero worker nodes, such as if you created a cluster without any worker nodes or manually removed all the worker nodes from the cluster. [Resize your worker pool](https://cloud.ibm.com/docs/containers?topic=containers-add_workersresize_pool) to add worker nodes to recover from a Warning state, and then [update the Calico node entries for your worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-zero_nodes_calico_failure). - A control plane operation for your cluster failed. View the cluster in the console or run ibmcloud ks cluster get --cluster <cluster_name_or_ID> to [check](https://cloud.ibm.com/docs/containers?topic=containers-debug_master) the Master Status for further debugging.
Check that your pod is Running. The pod does not run if no worker nodes with huge pages are available.

kubectl get pods
2. Log in to the pod.

kubectl exec -it <pod> /bin/sh
3. Verify that your pod can view the sizes of the huge pages.

ls /sys/kernel/mm/hugepages

Example output

hugepages-1048576kB hugepages-2048kB



8. Optional: Remove the enablement daemon set. Keep in mind that you must re-create the daemon set if you need to update, reload, replace, or add worker nodes with huge pages later.

kubectl -n kube-system delete daemonset hugepages-enablement
9. Repeat these steps whenever you update, reload, replace, or add worker nodes.



To troubleshoot worker nodes with huge pages, you can only reboot the worker node. The huge pages configuration does not persist across any other worker node life cycle operation, such as updating, reloading, replacing, or adding worker nodes. To remove the huge pages configuration from your cluster, you can update, reload, or replace all the worker nodes.





 Changing the Calico maximum transmission unit (MTU) 

Increase or decrease the Calico plug-in maximum transmission unit (MTU) to meet the network throughput requirements of your environment.

All VPC workers nodes support jumbo frames. However, on classic infrastructure, only bare metal workers support jumbo frames.

By default, the Calico network plug-in in your IBM Cloud Kubernetes Service cluster has an MTU of 1480 bytes. For most cases, this default MTU value provides sufficient throughput for packets that are sent and received in your network workloads. Review the following cases in which you might need to modify the default Calico MTU:



* Jumbo frames have an MTU value in the range of 1500 to 9000. To ensure that your cluster's pod network can use this higher MTU value, you can increase the Calico MTU to 20 bytes less than the jumbo frame MTU.