vania-janet/multiturn-rag-retrieval-data huggingface dataset:
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: myvol
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: myvol
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
iops: "300" required only for performance storage
storageClassName: ibmc-file-retain-bronze
Show more

Example stateful set with anti-affinity rule and delayed File Storage for Classic creation. The following example shows how to deploy NGINX as a stateful set with 3 replicas. The stateful set does not specify the region and zone where the File Storage for Classic is created. Instead, the stateful set uses an anti-affinity rule to ensure that the pods are spread across worker nodes and zones. Worker node anti-affinity is achieved by defining the app: nginx label. This label instructs the Kubernetes scheduler to not schedule a pod on a worker node if a pod with the same label already runs on this worker node. The topologykey: failure-domain.beta.kubernetes.io/zone label restricts this anti-affinity rule even more and prevents the pod to be scheduled on a worker node that is located in the same zone as a worker node that already runs a pod with the app: nginx label. For each stateful set pod, two PVCs are created as defined in the volumeClaimTemplates section, but the creation of the File Storage for Classic instances is delayed until a stateful set pod that uses the storage is scheduled. This setup is referred to as [topology-aware volume scheduling](https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibmc-file-bronze-delayed
parameters:
billingType: hourly
In addition, one public VPC load balancer is automatically created outside of your cluster in your VPC. When you enable private ALBs in your VPC cluster, a private VPC load balancer is also created.

IP addresses for ALBs in VPC clusters are not static and might change over time, therefore the VPC load balancers put the public or private IP address of your ALBs behind a static hostname. Separate hostnames are applied for public or private ALBs. Note that the ALB hostname is separate from the cluster's Ingress subdomain.

To find the hostname of an ALB in a VPC cluster, run ibmcloud ks ingress alb ls. Private hostnames are only listed if private ALBs are enabled.





 Worker node requirements for ALBs 

At least two worker nodes are required per zone in your cluster for ALBs to function with high availability and to receive periodic updates. Anti-affinity rules on ALB pods ensure that only one pod is scheduled to each worker node. When automatic updates are applied to ALB pods, the pod is reloaded. If you only have one worker node, and therefore one ALB pod, the pod does not automatically update to avoid traffic interruptions, and updates only apply when you manually delete the pod and reschedule a new one. Having at least two worker nodes per zone avoids this scenario.

Note that if a zone fails, you might experience intermittent failures in requests to the Ingress ALB in that zone.







 Default TLS certificate 

When you create a cluster, a default TLS certificate is created that you can use with the IBM-provided Ingress subdomain. You can specify the default TLS certificate, or a custom certificate that you provide, in the Ingress resource.

Consider using [Secrets Manager](https://cloud.ibm.com/docs/containers?topic=containers-secrets-mgr) to centrally manage and automatically update your Ingress subdomain certificates and other secrets.

Setting up Ingress with TLS certificates involves creating or importing secrets. If you want to use the IBM Cloud Ingress API to complete these steps, you must have a default Secrets Manager instance. Otherwise, you can use kubectl commands to copy your certificates.
If the usage is consistently over 80%, [add more worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workers) to the cluster to support the workloads.



5. Check for custom admission controllers that are installed in your cluster. Admission controllers often block required pods from running, which might make your worker nodes enter a critical state. If you have custom admission controllers, try removing them with oc delete. Then, check if the worker node issue resolves.

kubectl get mutatingwebhookconfigurations --all-namespaces

kubectl get validatingwebhookconfigurations --all-namespaces
6. If you configured [log forwarding](https://cloud.ibm.com/docs/containers?topic=containers-health), review the node-related logs from the following paths.

/var/log/containerd.log
/var/log/kubelet.log
/var/log/kube-proxy.log

/var/log/messages
7. Check that a workload deployment does not cause the worker node issue.



1. Taint the worker node with the issue.

oc taint node NODEIP ibm-cloud-debug-isolate-customer-workload=true:NoExecute
2. Make sure that you deleted any custom admission controllers as described in step 5.
3. Restart the worker node.



* Classic: Reload the worker node.

ibmcloud oc worker reload -c <cluster_name_or_ID> --worker <worker_ID>
* VPC: Replace the worker node.

ibmcloud oc worker replace -c <cluster_name_or_ID> --worker <worker_ID> --update



4. Wait for the worker node to finish restarting. If the worker node enters a healthy state, the issue is likely caused by a workload.
5. Schedule one workload at a time onto the worker node to see which workload causes the issue.
kube-dal10-crb20b637238ea471f8d4a8b881aae4962-w7 169.xx.xxx.xxx 10.xxx.xx.xxx b3c.4x16 provision_pending Ready dal10 1.26
kube-dal10-crb20b637238ea471f8d4a8b881aae4962-w8 169.xx.xxx.xxx 10.xxx.xx.xxx b3c.4x16 provision_pending Ready dal10 1.26







 Adding a zone to a worker pool 

You can span your classic cluster across multiple zones within one region by adding a zone to your existing worker pool.

When you add a zone to a worker pool, the worker nodes that are defined in your worker pool are provisioned in the new zone and considered for future workload scheduling. Red Hat OpenShift on IBM Cloud automatically adds the failure-domain.beta.kubernetes.io/region label for the region and the failure-domain.beta.kubernetes.io/zone label for the zone to each worker node. The Kubernetes scheduler uses these labels to spread pods across zones within the same region.

If you have multiple worker pools in your cluster, add the zone to all them so that worker nodes are spread evenly across your cluster. Note that when you add worker nodes to your cluster, the new worker nodes run the same major.minor version as the cluster master, but the latest worker node patch of that major.minor version.

Before you begin:



* To add a zone to your worker pool, your worker pool must be in a [multizone-capable zone](https://cloud.ibm.com/docs/openshift?topic=openshift-regions-and-zoneszones-mz). If your worker pool is not in a multizone-capable zone, consider [creating a new worker pool](https://cloud.ibm.com/docs/openshift?topic=openshift-add_workersadd_pool).
--size-per-zone WORKERS_PER_ZONE
: The number of workers that you want to have in each zone. This value is required, and must be 1 or greater. For more information, see [What is the smallest size cluster that I can make?](https://cloud.ibm.com/docs/containers?topic=containers-faqssmallest_cluster).

-q
: Optional: Do not show the message of the day or update reminders.



 Example worker-pool resize command 

ibmcloud ks worker-pool resize --cluster my_cluster --worker-pool my_pool --size-per-zone 3







 ibmcloud ks worker-pool rm 

Virtual Private Cloud

Classic infrastructure

Remove a worker pool from your cluster. All worker nodes in the pool are deleted. Your pods are rescheduled when you delete. To avoid downtime, be sure that you have enough workers to run your workload.

ibmcloud ks worker-pool rm --worker-pool WORKER_POOL --cluster CLUSTER [-q] [-f]

Minimum required permissions
: Operator platform access role for the cluster in IBM Cloud Kubernetes Service

Command options:

-p, --worker-pool WORKER_POOL
: Required: The name of the worker node pool that you want to remove.

-c, --cluster CLUSTER
: Required: The name or ID of the cluster that you want to remove the worker pool from.

-q
: Optional: Do not show the message of the day or update reminders.

-f
: Optional: Force the command to run with no user prompts.



 Example worker-pool rm command 

ibmcloud ks worker-pool rm --cluster my_cluster --worker-pool pool1







 ibmcloud ks worker-pool taint 

Set or remove Kubernetes taints for all the worker nodes in a worker pool. After you set up taints, pods without a matching toleration can't run on the worker pool.
- containerPort: 80
name: web
volumeMounts:
- name: myvol
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: myvol
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
iops: "300" required only for performance storage
storageClassName: ibmc-file-retain-bronze
Show more

Example stateful set with anti-affinity rule and delayed File Storage for Classic creation. The following example shows how to deploy NGINX as a stateful set with 3 replicas. The stateful set does not specify the region and zone where the File Storage for Classic is created. Instead, the stateful set uses an anti-affinity rule to ensure that the pods are spread across worker nodes and zones. Worker node anti-affinity is achieved by defining the app: nginx label. This label instructs the Kubernetes scheduler to not schedule a pod on a worker node if a pod with the same label already runs on this worker node. The topologykey: failure-domain.beta.kubernetes.io/zone label restricts this anti-affinity rule even more and prevents the pod to be scheduled on a worker node that is located in the same zone as a worker node that already runs a pod with the app: nginx label. For each stateful set pod, two PVCs are created as defined in the volumeClaimTemplates section, but the creation of the File Storage for Classic instances is delayed until a stateful set pod that uses the storage is scheduled. This setup is referred to as [topology-aware volume scheduling](https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibmc-file-bronze-delayed
parameters:
billingType: hourly
classVersion: "2"
iopsPerGB: "2"
You can add subnets by following the steps in [Configuring subnets for clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-subnets).
* Ensure that you have the [Writer or Manager IBM Cloud IAM service access role](https://cloud.ibm.com/docs/openshift?topic=openshift-userschecking-perms) for the default namespace.
* Ensure you have the required number of worker nodes:



* Classic clusters: If you restrict network traffic to edge worker nodes, ensure that at least two [edge worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-edgeedge) are enabled in each zone so that NLBs deploy uniformly.



* When cluster nodes are reloaded or when a cluster master update includes a new keepalived image, the load balancer virtual IP is moved to the network interface of a new node. When this occurs, any long-lasting connections to your load balancer must be re-established. Consider including retry logic in your application so that attempts to re-establish the connection are made quickly.



To set up an NLB 2.0 in a multizone cluster:



1. [Deploy your app to the cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_apps). Ensure that you add a label to your deployment in the metadata section of your configuration file. This custom label identifies all pods where your app runs to include them in the load balancing.
2. Create a load balancer service for the app that you want to expose to the public internet or a private network.



1. Create a service configuration file that is named, for example, myloadbalancer.yaml.
2. Define a load balancer service for the app that you want to expose. You can specify a zone, a VLAN, and an IP address.

apiVersion: v1
kind: Service
metadata:
name: myloadbalancer
annotations:
If you run privileged containers, run commands on the workers, or execute untrusted code, then you might be at risk.

Some change logs are for worker node fix packs, and apply only to worker nodes. You must [apply these patches](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_update) to ensure security compliance for your worker nodes. These worker node fix packs can be at a higher version than the master because some build fix packs are specific to worker nodes. Other change logs are for master fix packs, and apply only to the cluster master. Master fix packs might not be automatically applied. You can choose to [apply them manually](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_cluster_update). For more information about patch types, see [Update types](https://cloud.ibm.com/docs/containers?topic=containers-cs_versionsupdate_types).





 Version 1.21 change log 

Review the version 1.21 change log.



 Change log for worker node fix pack 1.21.14_1578, released 26 September 2022 

The following table shows the changes that are in the worker node fix pack 1.21.14_1578. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.21.14_1578

 Component Previous Current Description
drain_timeout_seconds: "120"
defaultcheck.json: 
{
"MaxUnavailablePercentage": 20
}
<check_name>: 
{
"MaxUnavailablePercentage": <value_in_percentage>,
"NodeSelectorKey": "<node_selector_key>",
"NodeSelectorValue": "<node_selector_value>"
}
Show more

drain_timeout_seconds
: Optional: The timeout in seconds to wait for the [drain](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) to complete. Draining a worker node safely removes all existing pods from the worker node and reschedules the pods onto other worker nodes in the cluster. Accepted values are integers in the range 1 - 180. The default value is 30.

defaultcheck.json
: When you update worker nodes in Satellite hosts, only 20% of the worker nodes in the cluster can be unavailable at a time.

MaxUnavailablePercentage
: The maximum number of nodes that are allowed to be unavailable for a specified label key and value, which is specified as a percentage. A worker node is unavailable during the deploying, reloading, or provisioning process. The queued worker nodes are blocked from updating if it exceeds any defined maximum unavailable percentages.

NodeSelectorKey
: The label key of the worker node for which you want to set a rule. You can set rules for the default labels that are provided by IBM, as well as on worker node labels that you created.

NodeSelectorValue
: The label value that the worker node must have to be considered for the rule that you define.
5. Create the ConfigMap in your cluster.

kubectl apply -f <filepath/configmap.yaml>
6. Verify that the ConfigMap is created.

kubectl get configmap --namespace kube-system
7. Update the worker nodes by listing them by ID.
What network traffic is allowed for my Classic cluster by default? 

All containers are protected by [predefined Calico network policy settings](https://cloud.ibm.com/docs/containers?topic=containers-network_policiesdefault_policy) that are configured on every worker node during cluster creation. By default, all outbound network traffic is allowed for all worker nodes. Inbound network traffic is blocked with the following exceptions:



* NodePort: The [Kubernetes NodePort range](https://kubernetes.io/docs/concepts/services-networking/service/nodeport) is opened by default so that you can expose apps with [NodePort services](https://cloud.ibm.com/docs/containers?topic=containers-nodeport). To block inbound network traffic on NodePorts in your cluster, see [Controlling inbound traffic to NLB or NodePort services](https://cloud.ibm.com/docs/containers?topic=containers-network_policiesblock_ingress).
* IBM monitoring ports: By default, IBM opens a few ports on your cluster so that network traffic can be monitored by IBM and for IBM to automatically install security updates for the Kubernetes master.



Access from the Kubernetes master to the worker node's kubelet is secured by a Konnectivity tunnel. For more information, see the [IBM Cloud Kubernetes Service architecture](https://cloud.ibm.com/docs/containers?topic=containers-service-arch).





 What is network segmentation and how can I set it up for a Classic cluster? 

Network segmentation describes the approach to divide a network into multiple subnetworks. You can group apps and related data to be accessed by a specific group in your organization. Apps that run in one subnetwork can't see or access apps in another subnetwork. Network segmentation also limits the access that is provided to an insider or third-party software and can limit the range of malicious activities.

IBM Cloud Kubernetes Service provides IBM Cloud VLANs that ensure quality network performance and network isolation for worker nodes.
What network traffic is allowed for my Classic cluster by default? 

All containers are protected by [predefined Calico network policy settings](https://cloud.ibm.com/docs/openshift?topic=openshift-network_policiesdefault_policy) that are configured on every worker node during cluster creation. By default, all outbound network traffic is allowed for all worker nodes. Inbound network traffic is blocked with the following exceptions:



* NodePort: The [Kubernetes NodePort range](https://kubernetes.io/docs/concepts/services-networking/service/nodeport) is opened by default so that you can expose apps with [NodePort services](https://cloud.ibm.com/docs/openshift?topic=openshift-nodeport). To block inbound network traffic on NodePorts in your cluster, see [Controlling inbound traffic to NLB or NodePort services](https://cloud.ibm.com/docs/openshift?topic=openshift-network_policiesblock_ingress).
* IBM monitoring ports: By default, IBM opens a few ports on your cluster so that network traffic can be monitored by IBM and for IBM to automatically install security updates for the Red Hat OpenShift master.



Access from the Red Hat OpenShift master to the worker node's kubelet is secured by a Konnectivity tunnel. For more information, see the [Red Hat OpenShift on IBM Cloud architecture](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architecture).





 What is network segmentation and how can I set it up for a Classic cluster? 

Network segmentation describes the approach to divide a network into multiple subnetworks. You can group apps and related data to be accessed by a specific group in your organization. Apps that run in one subnetwork can't see or access apps in another subnetwork. Network segmentation also limits the access that is provided to an insider or third-party software and can limit the range of malicious activities.
Portable private IPs are permanent, fixed IP addresses that can be used to access NLBs or ALBs over a private network. If you need more than 4 IPs for private NLBs or ALBs, see [Adding portable IP addresses](https://cloud.ibm.com/docs/containers?topic=containers-subnetsadding_ips). Note that these IP addresses are intended for use in IBM Cloud Kubernetes Service. Do not use these IP addresses for other purposes outside of IBM Cloud Kubernetes Service.









 Finding subnets provisioned in your account 

To see all the subnets provisioned in all resource groups of your account, run ibmcloud ks subnets --provider classic. To see the portable public and portable private subnets that are bound to one cluster, you can run ibmcloud ks cluster get --cluster <cluster_name_or_ID> --show-resources and look for the Subnet VLANs section.

In IBM Cloud Kubernetes Service, VLANs have a limit of 40 subnets. If you reach this limit, first check to see whether you can [reuse subnets in the VLAN to create new clusters](https://cloud.ibm.com/docs/containers?topic=containers-subnetssubnets_custom). If you need a new VLAN, order one by [contacting IBM Cloud support](https://cloud.ibm.com/docs/vlans?topic=vlans-ordering-premium-vlansordering-premium-vlans). Then, [create a cluster](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_cluster_create) that uses this new VLAN.

Do the IP addresses for my worker nodes change?
: Your worker node is assigned an IP address on the public or private VLANs that your cluster uses. After the worker node is provisioned, the worker node IP address persists across reboot and update operations, but the worker node IP address changes after a replace operation. Additionally, the private IP address of the worker node is used for the worker node identity in most kubectl commands.
To check whether a VRF is already enabled, use the ibmcloud account show command. If you can't or don't want to enable VRF, enable [VLAN spanning](https://cloud.ibm.com/docs/vlans?topic=vlans-vlan-spanningvlan-spanning). When a VRF or VLAN spanning is enabled, the NLB 1.0 can route packets to various subnets in the account.
* Ensure you have the [Writer or Manager IBM Cloud IAM service access role](https://cloud.ibm.com/docs/containers?topic=containers-userschecking-perms) for the default namespace.
* Ensure you have the required number of worker nodes:



* Classic clusters: If you restrict network traffic to edge worker nodes, ensure that at least two [edge worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-edgeedge) are enabled in each zone so that NLBs deploy uniformly.
* Gateway-enabled classic clusters: Ensure that at least two gateway worker nodes in the gateway worker pool are enabled in each zone so that NLBs deploy uniformly.



* When cluster nodes are reloaded or when a cluster master update includes a new keepalived image, the load balancer virtual IP is moved to the network interface of a new node. When this occurs, any long-lasting connections to your load balancer must be re-established. Consider including retry logic in your application so that attempts to re-establish the connection are made quickly.



To set up an NLB 1.0 service in a multizone cluster:



1. [Deploy your app to the cluster](https://cloud.ibm.com/docs/containers?topic=containers-deploy_appapp_cli). Ensure that you add a label in the metadata section of your deployment configuration file. This custom label identifies all pods where your app runs to include them in the load balancing.
2. Create a load balancer service for the app that you want to expose to the public internet or a private network.



1. Create a service configuration file that is named, for example, myloadbalancer.yaml.
2.
Activity Tracker event N/A N/A Now, the containers-kubernetes.version.update event is sent to Activity Tracker when a master fix pack update is initiated for a cluster. 
 Cluster health image v1.1.18 v1.1.19 Updated image for [CVE-2020-28851](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28851). 
 IBM Cloud File Storage for Classic plug-in and monitor 388 389 Updated to use Go version 1.15.8. 
 Load balancer and load balancer monitor for IBM Cloud Provider 1165 1274 Fixed a bug that might cause version 2.0 network load balancers (NLBs) to crash and restart on load balancer updates. 
 Red Hat OpenShift 3.11.380 3.11.394 See the [Red Hat OpenShift release notes](https://docs.openshift.com/container-platform/3.11/release_notes/ocp_3_11_release_notes.htmlocp-3-11-394). 







 Change log for worker node fix pack 3.11.404_1584_openshift, released 29 March 2021 

The following table shows the changes that are in the worker node fix pack 3.11.404_1584_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 3.11.394_1582_openshift

 Component Previous Current Description 

 Red Hat OpenShift node 3.11.394 3.11.404 See the [Red Hat OpenShift release notes](https://docs.openshift.com/container-platform/3.11/release_notes/ocp_3_11_release_notes.htmlocp-3-11-404).