vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Additionally, if multiple app instances are deployed in one zone, the NLB routes the requests between the app pods in the zone.
4. When the app returns a response packet, it uses the IP address of the worker node where the NLB that forwarded the client request exists. The NLB then sends the response packet to the client.









 Components and architecture of an NLB 2.0 

The NLB 2.0 is a Layer 4 load balancer that uses the Linux kernel's IP Virtual Server (IPVS). The NLB 2.0 supports TCP and UDP, runs in front of multiple worker nodes, and uses IP over IP (IPIP) tunneling to distribute traffic that arrives to a single NLB IP address across those worker nodes.

Mixed protocol load balancer services are not supported in IBM Cloud Kubernetes Service. You cannot specify both TCP and UDP ports your in load balancer definition.



 Traffic flow in a single-zone cluster 

The following diagram shows how an NLB 2.0 directs communication from the internet to an app in a single zone cluster.

Zoom

![Expose an app in IBM Cloud Kubernetes Service by using a version 2.0 NLB](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/containers/images/cs_loadbalancer_ipvs_planning.png)

Figure 4. Expose an app in IBM Cloud Kubernetes Service by using a version 2.0 NLB



1. A client request to your app uses the public IP address of your NLB and the assigned port on the worker node. In this example, the NLB has a virtual IP address of 169.61.23.130 and runs on the worker node that has the 10.73.13.25 private IP address. Note that if you [create a DNS subdomain](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancer_hostname) for your NLB, users can access your app through the NLB's subdomain instead.
: Now, you can create GPU bare metal worker nodes in your Red Hat OpenShift on IBM Cloud version 4 clusters in select zones. GPU machines are suited for high-performance computing, machine learning, or 3D applications that you might have, such as [IBM Maximo Visual Inspection](https://www.ibm.com/products/maximo/remote-monitoring).
: To get started with GPU-accelerated workloads.



* Create a [cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters) or [worker pool](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersadd_pool) that uses a GPU bare metal flavor.
* [Install the Node Feature Discovery and NVIDIA GPU operators for your cluster version](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html).
* Verify that you can run GPU workloads by [deploying a sample job](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy_appgpu_app).



Master versions
A label or name is assigned to this key and is used by Key Protect in PKCS#11 API to refer to the master key.
* Signing key label: A label or name for a key in a partition, which is used for data authentication such as data signing and verification.
* Import key label: Key Protect supports importing Bring Your Own Key (BYOK) by a DES3 encryption Key. A label or name is assigned to this key.
* Secure import key label prefix (TKEK): Key Protect supports a secure mechanism to Bring Your Own Key (BYOK) into an HSM. The prefix used by the HSM for these keys must be known to Key Protect.
* Activity Tracker ingestion key: To receive audit logs, you must create an IBM Cloud Activity Tracker instance and an ingestion key.



There are two additional credentials you must share before your service can be deployed, the HSM client cert and the HSM client key. These credentials enable NTLS communications used by the Thales HSM for exchanges between the HSM and Key Protect running on your worker node. You must create and register with the HSM a TLS certificate for the worker node (client) that will connect to the HSM and provide the client certificate and key that Key Protect uses to communicate with the HSM. The exact instructions to share these credentials are communicated as part of your conversations with your service representative.

Make a note of these values for both HSMs before attempting to [deploy Key Protect](https://cloud.ibm.com/docs/key-protect?topic=key-protect-satellite-ui-deploy).
If multiple app instances are deployed in the cluster, the NLB routes the requests between the app pods.
3. When the app returns a response packet, it uses the IP address of the worker node where the NLB that forwarded the client request exists. The NLB then sends the response packet to the client.







 Traffic flow in a multizone cluster 

The following diagram shows how a network load balancer (NLB) 1.0 directs communication from the internet to an app in a multizone cluster.

Zoom

![Use an NLB 1.0 to load balance apps in a multizone cluster](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_loadbalancer_planning_multizone.png)

Figure 2. Use an NLB 1.0 to load balance apps in a multizone cluster



1. A request to your app uses the [DNS subdomain](https://cloud.ibm.com/docs/openshift?topic=openshift-loadbalancer_hostname) for your NLBs. You can also access the NLB in each zone by using its public IP address and port on the worker node. Note that by default, each NLB 1.0 is set up in one zone only. To achieve high availability, you must deploy an NLB 1.0 in every zone where you have app instances.
2. A DNS system service resolves the subdomain to the portable public IP address of one of the NLBs and its assigned port on the worker node. Requests are handled by the NLBs in various zones in a round-robin cycle.
3. The NLB receives the request and forwards it to the private IP address of the app pod over the private network. The source IP address of the request package is changed to the public IP address of the worker node where the NLB pod runs. Each NLB routes requests to the app instances in its own zone and to app instances in other zones.
Allow worker nodes to be created in your cluster. Allow ALL Any <br><br> * <br><br><br> 161.26.0.0/16 <br><br> * <br><br><br> Set to top 
 Allow worker nodes to communicate with other IBM Cloud services that support private cloud service endpoints, and in clusters that run Kubernetes version 1.19 or earlier, with the cluster master through the private cloud service endpoint. Allow ALL Any <br><br> * <br><br><br> 166.8.0.0/14 <br><br> * <br><br><br> After 1 
 Multizone clusters: Allow worker nodes in one subnet to communicate with the worker nodes in all other subnets within the cluster. Create one rule for each subnet that you want to connect to. Allow ALL Any <br><br> * <br><br><br> Other subnet's CIDR <br><br> * <br><br><br> After 2 
 Allow incoming traffic requests to apps that run on your worker nodes. Allow TCP Any 30000 - 32767 Any Any After 3 
 To expose apps by using load balancers or Ingress, allow traffic through VPC load balancers. Allow TCP Any 443 Any Any After 4 
 * Allow access from the Kubernetes control plane IP addresses that are used to health check and report the overall status of your Ingress components. Create one rule for each [control plane CIDR for the region where your cluster is located](https://github.com/IBM-Cloud/kube-samples/tree/master/control-plane-ips). Allow TCP Any 80 Each [control plane CIDR for the region where your cluster is located](https://github.com/IBM-Cloud/kube-samples/tree/master/control-plane-ips). Any After 5 
 Deny all other traffic that does not match the previous rules. Deny ALL Any <br><br> * <br><br><br> Any <br><br> * <br><br><br> Set to bottom
OpenVPN client 2.4.6-r3-IKS-463 2.5.4-r0-IKS-556 Update base image to alpine 3.15 to address CVEs, no longer set the --compress config option, updated scripts. 
 OpenVPN server 2.4.6-r3-IKS-462 2.5.4-r0-IKS-555 Update base image to alpine 3.15 to address CVEs, no longer set the --compress config option, updated scripts. 
 OpenVPN Operator image v1.3.9 v1.4.0 Updated OpenVPN to version 2.5.4-r0. Remove deprecated compression config option from VPN server. Set cipher explicitly to default value to avoid warning message in logs. 
 Portieris admission controller v0.12.1 v0.12.2 See the [Portieris admission controller release notes](https://github.com/IBM/portieris/releases/tag/v0.12.2) 
 IBM Cloud RBAC Operator 3430e03 0fc9949 Updated universal base image (UBI) to the 8.5-218 version to resolve CVEs. Updated to use Go version 1.16.13. 
 IBM Cloud File Storage for Classic plug-in and monitor 402 404 Updated universal base image (UBI) to the 8.5-218 version to resolve CVEs. Updated to use Go version 1.16.13. 







 Change log for worker node fix pack 4.8.26_1541_openshift, released 18 January 2022 

The following table shows the changes that are in the worker node fix pack 4.8.26_1541_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.8.24_1540_openshift

 Component Previous Current Description
IBM Cloud Provider v1.8.11-126 v1.8.13-176 Updated to support Kubernetes 1.8.13 release. 
 OpenVPN client N/A N/A Added livenessProbe to the OpenVPN client vpn deployment that runs in the kube-system namespace. 







 Change log for worker node fix pack 1.8.11_1512, released 11 June 2018 



Table 1. Changes since version 1.8.11_1511

 Component Previous Current Description 

 Kernel update 4.4.0-116 4.4.0-127 New worker node images with kernel update for [CVE-2018-3639](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3639) 







 Change log for worker node fix pack 1.8.11_1511, released 18 May 2018 



Table 1. Changes since version 1.8.11_1510

 Component Previous Current Description 

 Kubelet N/A N/A Fix to address a bug that occurred if you used the block storage plug-in. 







 Change log for worker node fix pack 1.8.11_1510, released 16 May 2018 



Table 1. Changes since version 1.8.11_1509

 Component Previous Current Description 

 Kubelet N/A N/A The data that you store in the kubelet root directory is now saved on the larger, secondary disk of your worker node machine. 







 Change log for 1.8.11_1509, released 19 April 2018 



Table 1. Changes since version 1.8.8_1507

 Component Previous Current Description 

 Kubernetes v1.8.8 v1.8.11 See the [Kubernetes release notes](https://github.com/kubernetes/kubernetes/releases/tag/v1.8.11).
* Obtain the packages that are required to perform cluster updates.
* Access Red Hat’s software as a service page to perform subscription management.



In the reference architecture, the following components are installed and configured in the build process:



* Bastion node - This RHEL VM acts as a "jump-server" on the overlay network to enable the build process. It is accessed by using SSH through the private network. It also hosts a webserver to help the build process of the cluster.
* Bootstrap node - As each node in the cluster requires information about the cluster when it is provisioned, a temporary bootstrap node is used. The bootstrap node creates the control plane nodes that make up the control plane. The control plane nodes then create the worker nodes. After the cluster initializes, the bootstrap node can be destroyed.
* Control-plane nodes - These nodes run services that are required to control the Kubernetes cluster. They contain more than just the Kubernetes services for managing the cluster. The terms "primary" and "control-plane" are used interchangeably.
* Compute nodes - In a Kubernetes cluster, the compute nodes are where the workloads are run. The compute nodes advertise their capacity to the control-plane nodes.
* DNS - A correct DNS setup is imperative for a functioning Red Hat OpenShift cluster. The vCenter Server instance AD DNS server to host the required DNS records.
* Load-balancer - An NSX ESG load-balancer service is used to front end the Red Hat OpenShift APIs, both internal and external, and the Red Hat OpenShift router. The load balancer is configured so that Port 6443 and 22623 point to the bootstrap and control plane nodes, while ports 80 and 443 are configured to point to the worker nodes.
* Webserver - A web server is needed to hold the ignition configurations and installation images for the installation of RHEL CoreOS. NGINX is installed on the bastion node to provide this function.
A DNS system service resolves the subdomain to the portable public IP address of one of the NLBs and its assigned port on the worker node. Requests are handled by the NLBs in various zones in a round-robin cycle.
3. The NLB receives the request and forwards it to the private IP address of the app pod over the private network. The source IP address of the request package is changed to the public IP address of the worker node where the NLB pod runs. Each NLB routes requests to the app instances in its own zone and to app instances in other zones. Additionally, if multiple app instances are deployed in one zone, the NLB routes the requests between the app pods in the zone.
4. When the app returns a response packet, it uses the IP address of the worker node where the NLB that forwarded the client request exists. The NLB then sends the response packet to the client.









 Components and architecture of an NLB 2.0 

The NLB 2.0 is a Layer 4 load balancer that uses the Linux kernel's IP Virtual Server (IPVS). The NLB 2.0 supports TCP and UDP, runs in front of multiple worker nodes, and uses IP over IP (IPIP) tunneling to distribute traffic that arrives to a single NLB IP address across those worker nodes.

Mixed protocol load balancer services are not supported in IBM Cloud Kubernetes Service. You cannot specify both TCP and UDP ports your in load balancer definition.



 Traffic flow in a single-zone cluster 

The following diagram shows how an NLB 2.0 directs communication from the internet to an app in a single zone cluster.

Zoom

![Expose an app in Red Hat OpenShift on IBM Cloud by using a version 2.0 NLB](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_loadbalancer_ipvs_planning.png)

Figure 4. Expose an app in Red Hat OpenShift on IBM Cloud by using a version 2.0 NLB
: If Cluster is set, DSR is implemented only from the worker node that the VPC NLB initially forwards the incoming request to. After the incoming request arrives, the request is forwarded to a worker node that contains the app pod, which may be in a different zone. The response from the app pod is sent to the original worker node, and that worker node uses DSR to send the response directly back to the client, bypassing the VPC NLB. This option also configures TCP health checks. For UDP load balancers, the service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-health-check-udp must be set with a TCP port value. For more information, see [Configuring TCP health checks for UDP load balancers](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-lbaasvpc_lb_health_udp).
3. Create the Kubernetes LoadBalancer service in your cluster.

oc apply -f <filename>.yaml -n <namespace>
4. Verify that the Kubernetes LoadBalancer service is created successfully in your cluster. When the service is created, the LoadBalancer Ingress field is populated with an external IP address that is assigned by the VPC NLB.

The VPC NLB takes a few minutes to provision in your VPC. The external IP address of your Kubernetes LoadBalancer service might be pending until the VPC NLB is fully provisioned.

oc describe svc myloadbalancer -n <namespace>

Example CLI output for a public LoadBalancer service:

NAME: myvpcnlb
Namespace: default
Labels: <none>
Annotations: service.kubernetes.io/ibm-load-balancer-cloud-provider-enable-features: nlb
Selector: app=echo-server
Type: LoadBalancer
IP: 172.21.204.12
LoadBalancer Ingress: 169.XXX.XXX.XXX
Change log for worker node fix pack 1.16.3_1518, released 25 November 2019 

The following table shows the changes that are in the worker node fix pack 1.16.3_1518.



Changes since version 1.16.2_1515

 Component Previous Current Description 

 containerd v1.3.0 v1.3.1 See the [containerd release notes](https://github.com/containerd/containerd/releases/tag/v1.3.1). Update resolves [CVE-2019-16884](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-16884) and [CVE-2019-17596](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17596). 
 Kubernetes v1.16.2 v1.16.3 See the [Kubernetes release notes](https://github.com/kubernetes/kubernetes/releases/tag/v1.16.3).
Load balancer and load balancer monitor for IBM Cloud Provider 146 147 Image updated for [CVE-2019-14697](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14697). 
 OpenVPN client 2.4.6-r3-IKS-13 2.4.6-r3-IKS-116 Image updated for [CVE-2019-14697](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14697). 
 OpenVPN server 2.4.6-r3-IKS-25 2.4.6-r3-IKS-115 Image updated for [CVE-2019-14697](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14697). 







 Change log for worker node fix pack 1.12.10_1561, released 5 August 2019 

The following table shows the changes that are in the worker node fix pack 1.12.10_1561.



Changes since version 1.12.10_1560

 Component Previous Current Description
Multizone clusters: Add rules to your ACL to allow worker nodes in one subnet to communicate with the worker nodes in all other subnets within the cluster. The subnet for which you create the ACL rule is defined as 0.0.0.0/0 and you use the CIDRs of the other subnets as your destination CIDR. Make sure to create one inbound and one outbound rule for each of the subnets that you want to connect to.

ibmcloud is network-acl-rule-add $acl_id allow outbound all 0.0.0.0/0 <other_zone_subnet_CIDR> --name allow-workers-outbound

ibmcloud is network-acl-rule-add $acl_id allow inbound all <other_zone_subnet_CIDR> 0.0.0.0/0 --name allow-workers-inbound
6. Create rules to allow inbound traffic from and outbound traffic to the 161.26.0.0/16 and 166.8.0.0/14 IBM Cloud private subnets. The 161.26.0.0/16 rules allow you to create worker nodes in your cluster. The 166.8.0.0/14 rules allow worker nodes to communicate with other IBM Cloud services that support private cloud service endpoints, and in clusters that run Kubernetes version 1.19 or earlier, with the cluster master through the private cloud service endpoint.

Need to connect your worker nodes to IBM Cloud services that support only public cloud service endpoints? [Attach a public gateway to the subnet](https://cloud.ibm.com/kubernetes/clusters) so that worker nodes can connect to a public endpoint outside of your cluster. Then, create inbound and outbound rules to allow ingress from and egress to the services' public cloud service endpoints.

ibmcloud is network-acl-rule-add $acl_id allow outbound all 0.0.0.0/0 161.26.0.0/16 --name allow-ibm-private-network-outbound1
: You can now create VPC clusters in the Sao Paulo, Brazil [location](https://cloud.ibm.com/docs/containers?topic=containers-regions-and-zones).

VPC disk encryption on worker nodes
: Now, you can manage the encryption for the disk on your VPC worker nodes. For more information, see [VPC worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-encryptionworker-encryption-vpc).





 30 August 2021 

Review the release notes for 30 August 2021.

Worker node fix pack update
: Change log documentation is available for version [1.17.17_1568](https://cloud.ibm.com/docs/containers?topic=containers-117_changelog11717_1568), [1.18.20_1563](https://cloud.ibm.com/docs/containers?topic=containers-118_changelog11820_1563), [1.19.14_1558](https://cloud.ibm.com/docs/containers?topic=containers-changelog_11911914_1558), [1.20.10_1551](https://cloud.ibm.com/docs/containers?topic=containers-changelog_12012010_1551), and [1.21.4_1529](https://cloud.ibm.com/docs/containers?topic=containers-changelog_12112104_1529).





 25 August 2021 

New! Create a cluster with a template
: No longer do you have to manually specify the networking and worker node details to create a cluster, or enable security integrations such as logging and monitoring after creation. Instead, you can try out the technical preview to create a multizone cluster with nine worker nodes and encryption, logging, and monitoring already enabled. For more information, see [Creating a cluster by using an IBM Cloud Schematics template](https://cloud.ibm.com/docs/openshift?topic=openshift-templates&interface=ui).

IBM Cloud Object Storage plug-in
If you delete a VPC subnet that your cluster used, any load balancers that use IP addresses from the subnet might experience issues, and you might be unable to create new load balancers.









 Restricting public network traffic to a subnet with a public gateway 

Improve the security of your IBM Cloud® Kubernetes Service cluster by allowing fewer worker nodes to have external access through a VPC subnet public gateway.

If pods on your worker nodes need to connect to a public external endpoint, you can attach a public gateway to the subnet that those worker nodes are on. For example, your VPC cluster can automatically connect to other [IBM Cloud services that support private cloud service endpoints](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint), such as IBM Cloud Container Registry. However, if you need to access IBM Cloud services that support only public cloud service endpoints, you can attach a public gateway to the subnet so that your pods can send requests over the public network.

You can isolate this network traffic in your cluster by attaching a public gateway to only one subnet in your cluster. Then, you can use app affinity to deploy app pods that require access to external endpoints to only the subnet with an attached public gateway.

In VPC clusters, a subnet is limited to one zone. When you attach a public gateway to only one subnet, and schedule app pods that require public access to only worker nodes on that subnet, these pods are isolated to one zone in your cluster.



1. Target the region of the VPC that your cluster is deployed to.

ibmcloud target -r <region>
2. Check whether you have a public gateway in a zone where you have worker nodes. Within one VPC, you can create only one public gateway per zone, but that public gateway can be attached to multiple subnets within the zone.

ibmcloud is public-gateways

Example output

ID Name VPC Zone Floating IP Created Status Resource group