vania-janet/multiturn-rag-retrieval-data huggingface dataset:
: Updated the IBM Cloud Kubernetes Service CLI plug-in change log page for the [release of version 1.0.0](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_changelog10). This version contains permanent syntax and behavior changes that are not compatible with earlier versions, so before you update be sure to follow the guidance in [Using version 1.0 of the plug-in](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_changelogchangelog_beta).

Installing SGX drivers
: Added a topic for [installing SGX drivers and platform software on SGX-capable worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-add_workersinstall-sgx).

Sizing workloads
: Enhanced the topic with a [How do I monitor resource usage and capacity in my cluster?](https://cloud.ibm.com/docs/containers?topic=containers-strategysizing_manage) FAQ.
: sticky-cookie-services annotation.
: Added the secure and httponly parameters to the [sticky-cookie-services annotation](https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotationssession-affinity-cookies).

Version change logs
: Master and worker node patch updates are available.
Change log for master fix pack 1.17.11_1537, released 18 August 2020 

The following table shows the changes that are in the master fix pack patch update 1.17.11_1537. Master patch updates are applied automatically.



Changes since version 1.17.9_1535

 Component Previous Current Description 

 Cluster health image v1.1.8 v1.1.9 Updated to use Go version 1.13.13. 
 etcd v3.4.9 v3.4.10 See the [etcd release notes](https://github.com/etcd-io/etcd/releases/v3.4.10). 
 GPU device plug-in and installer 8c24345 bacb9e1 Updated image for [CVE-2020-14039](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14039) and [CVE-2020-15586](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15586). 
 IBM Cloud File Storage for Classic plug-in and monitor 376 377 Fixed a bug that prevents persistent volume claim (PVC) creation failures from being retried. 
 Key Management Service provider v1.0.0 v1.0.1 Updated image for [CVE-2020-15586](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15586). 
 Kubernetes v1.17.9 v1.17.11 See the [Kubernetes release notes](https://github.com/kubernetes/kubernetes/releases/tag/v1.17.11). 
 Kubernetes add-on resizer 1.8.5 1.8.11 See the [Kubernetes add-on resizer release notes](https://github.com/kubernetes/autoscaler/releases/tag/addon-resizer-1.8.11).
Excluded ingress status from cluster status calculation. 
 IBM Cloud Controller Manager v1.20.15-17 v1.20.15-19 Updated to support the Kubernetes 1.20.15 release. 
 IBM Cloud RBAC Operator dc1725a 778ef2b Updated to Go version 1.18.6. 
 Key Management Service provider v2.5.9 v2.5.10 Updated Go dependencies and to Go version 1.19.2. 
 OpenVPN Operator image v1.4.9 v1.4.10 Updated ansible operator base image to v1.24.0 to resolve CVEs. 







 Change log for worker node fix pack 4.7.59_1585_openshift, released 25 October 2022 

The following table shows the changes that are in the worker node fix pack 4.7.59_1585_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.7.59_1584_openshift

 Component Previous Current Description 

 RHEL 7 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud N/A N/A N/A 







 Change log for worker node fix pack 4.7.59_1584_openshift, released 10 October 2022 

The following table shows the changes that are in the worker node fix pack 4.7.59_1584_openshift. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 4.7.59_1583_openshift

 Component Previous Current Description 

 RHEL 7 Packages N/A N/A N/A 
 RHEL 8 Packages N/A N/A N/A 
 Red Hat OpenShift on IBM Cloud N/A N/A N/A 







 Change log for master fix pack 4.7.59_1582_openshift, released 26 September 2022 

The following table shows the changes that are in the master fix pack 4.7.59_1582_openshift.
etcd Master v3.3.18 v3.3.20 See the [etcd release notes](https://github.com/etcd-io/etcd/releases/v3.3.20). 
 Gateway-enabled cluster controller Master 1045 1082 Updated image for [CVE-2020-1967](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1967). 
 GPU device plug-in and installer Master 8c6538f 8b02302 Updated image for [CVE-2020-1967](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1967). 
 IBM Cloud File Storage for Classic plug-in and monitor Master 358 373 Updated image for [CVE-2020-1967](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-1967) and [CVE-2020-11655](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-11655). 
 IBM Cloud Provider Master v1.15.11-274 v1.15.12-316 Updated to support the Kubernetes 1.15.2 release. 
 Kubernetes Both v1.15.11 v1.15.12 See the [Kubernetes release notes](https://github.com/kubernetes/kubernetes/releases/tag/v1.15.12). The master update resolves CVE-2020-8555 (see the [IBM security bulletin](https://www.ibm.com/support/pages/node/6220220)). 
 Kubernetes Metrics Server Master N/A N/A Increased the CPU per node for the metrics-server container to improve availability of the metrics server API service for large clusters.
Endpoint Security Mitigation Kubernetes cluster role system:aggregate-to-edit has removed endpoints permissions as a security mitigation for [CVE-2021-25740](https://nvd.nist.gov/vuln/detail/CVE-2021-25740). If your cluster does not require any customizations to the system:aggregate-to-edit cluster role, besides removing the endpoints permission, allow Kubernetes to reconcile the permissions by running the kubectl annotate --overwrite clusterrole/system:aggregate-to-edit rbac.authorization.kubernetes.io/autoupdate=true command. Subsequent cluster master operations (for example, ibmcloud ks cluster master refresh) will then ensure the permissions are reconciled by Kubernetes. 
 Unsupported: kubectl autoscale removes --generator option The kubectl austoscale no longer uses the deprecated --generator option. If your scripts rely on this option, update them. 
 Unsupported: kubectl create deployment removes --generator option The kubectl create deployment command no longer uses the deprecated --generator option. If your scripts rely on this option, update them. 
 system:aggregate-to-edit write access for Endpoints API The system:aggregate-to-edit role no longer includes write access to the Endpoints API. Existing clusters that are upgraded to Kubernetes 1.22 are not impacted. However, in new Kubernetes 1.22 clusters, the Editor and Administrator roles don't have write access to the Endpoints API. For more information on retaining this access in newly created 1.22 clusters, see [Write access for Endpoints](https://kubernetes.io/docs/reference/access-authn-authz/rbac/write-access-for-endpoints). This update is a mitigation for [CVE-2021-25740](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25740).
Kubernetes N/A N/A N/A 
 Containerd 1.7.0-rc.1 1.7.0-rc.3 For more information, see the [change log](https://github.com/containerd/containerd/releases/tag/v1.7.0-rc.3). 







 Change log for master fix pack 1.26.1_1524, released 2 March 2023 

The following table shows the changes that are in the master fix pack 1.26.1_1524. Master patch updates are applied automatically.



Changes since version 1.26.1_1519

 Component Previous Current Description 

 Calico v3.24.5 v3.25.0 See the [Calico release notes](https://docs.tigera.io/archive/v3.24/release-notes/v3245). 
 Cluster health image v1.3.15 v1.3.16 Updated Go dependencies and to Go version 1.19.6. Updated universal base image (UBI) to resolve CVEs. 
 CoreDNS 1.10.0 1.10.1 See the [CoreDNS release notes](https://coredns.io/tags/notes/). 
 etcd v3.5.6 v3.5.7 See the [etcd release notes](https://github.com/etcd-io/etcd/releases/v3.5.7). 
 IBM Calico extension 1305-amd64 1308-amd64 Updated universal base image (UBI) to resolve [CVE-2022-47629](https://nvd.nist.gov/vuln/detail/CVE-2022-47629). 
 IBM Cloud Block Storage driver and plug-in v2.3.6 v2.3.7 Updated universal base image (UBI) to resolve CVEs. 
 IBM Cloud Controller Manager v1.26.1-2 v1.26.1-16 Updated Go dependencies. Updated k8s.io/utils digest to a5ecb01.
: Updated the [nginx-ingress build to 651 and the ingress-auth build to 423](https://cloud.ibm.com/docs/containers?topic=containers-cluster-add-ons-changelogkube_ingress_changelog) for the IBM Cloud Kubernetes Service Ingress image.

Istio add-on
: Version [1.7.2](https://cloud.ibm.com/docs/containers?topic=containers-istio-changelog172) of the Istio managed add-on is released.





 22 September 2020 

Unsupported: Kubernetes version 1.15
: Clusters that run version 1.15 are unsupported. To continue receiving important security updates and support, you must [update the cluster to a supported version](https://cloud.ibm.com/docs/containers?topic=containers-cs_versionsprep-up) immediately.





 21 September 2020 

Versions
: Master fix pack update change log documentation is available.
: Kubernetes version [1.18.9_1528](https://cloud.ibm.com/docs/containers?topic=containers-118_changelog1189_1528), [1.17.12_1540](https://cloud.ibm.com/docs/containers?topic=containers-117_changelog11712_1540), and [1.16.15_1547](https://cloud.ibm.com/docs/containers?topic=containers-116_changelog11615_1547).

Versions
: Worker node fix pack update change log documentation is available.
Kubernetes 1.21.7 1.21.9 For more information, see the [change log](https://github.com/kubernetes/kubernetes/releases/tag/v1.21.9). 







 Change log for master fix pack 1.21.9_1547, released 26 January 2022 



Changes since version 1.21.7_1541

 Component Previous Current Description 

 Calico N/A N/A Changed to improve Calico availability during updates. 
 Cluster health image v1.2.20 v1.2.21 Updated to use Go version 1.17.5, updated Go dependencies and golangci-lint 
 GPU device plug-in and installer c9bfc8c eefc4ae Updated universal base image (UBI) to the 8.5-218 version to resolve CVEs. Updated to use Go version 1.16.13. 
 IBM Calico extension 900 923 Updated universal base image (UBI) to the 8.5-218 version to resolve CVEs. Updated to use Go version 1.16.13. 
 IBM Cloud Controller Manager v1.21.7-3 v1.21.9-1 Updated to support the Kubernetes 1.21.9 release and to use Go version 1.16.12. 
 IBM Cloud File Storage for Classic plug-in and monitor 402 404 Updated universal base image (UBI) to the 8.5-218 version to resolve CVEs. Updated to use Go version 1.16.13. 
 IBM Cloud RBAC Operator 3430e03 0fc9949 Updated universal base image (UBI) to the 8.5-218 version to resolve CVEs. Updated to use Go version 1.16.13. 
 Key Management Service provider v2.3.12 v2.3.13 Updated Go dependencies and golangci-lint 
 Konnectivity agent v0.0.26_282_iks v0.0.27_309_iks See the [Konnectivity release notes](https://github.com/kubernetes-sigs/apiserver-network-proxy/releases/tag/v0.0.27).
March 2022 



 30 March 2022 

Cluster autoscaler add-on
: Version [1.1.0_475](https://cloud.ibm.com/docs/containers?topic=containers-ca_changelog) is available.

Kubernetes Service 1.20 end of support date change
: The end of support date of Kubernetes Service 1.20 is now 15 June 2022. The [Kubernetes version information and update actions](https://cloud.ibm.com/docs/containers?topic=containers-cs_versionscs_versions_available) page has been updated with the new date.

Master fix pack update
: Kubernetes [1.23.5_1523](https://cloud.ibm.com/docs/containers?topic=containers-changelog_1231235_1523), [1.22.8_1545](https://cloud.ibm.com/docs/containers?topic=containers-changelog_1221228_1545), [1.21.11_1553](https://cloud.ibm.com/docs/containers?topic=containers-changelog_12112111_1553), and [1.20.15_1574](https://cloud.ibm.com/docs/containers?topic=containers-changelog_12012015_1574).





 28 March 2022 

Worker node fix pack update.
* [30 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march3021)



* add-on change logs
* Master versions



* [29 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march2921)



* Worker node versions



* [25 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march2521)



* CIS Kubernetes Benchmark
* CRI-O container runtime
* Certificate Manager instances



* [22 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march2221)



* Ingress updates



* [17 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-mar1721)



* Deprecated events to be replaced for Activity Tracker
* Deprecated fields across events



* [16 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march1621)



* Ingress ALB change log



* [12 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march1221)



* Worker node versions



* [9 March 2021](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift-relnotesopenshift-march0921)



* Cluster autoscaler add-on
"masterURL": "https://c2.private.us-south.containers.cloud.ibm.com:30422",
"ingress": {
...}
5. To use a private cloud service endpoint, you must first expose the private cloud service endpoint by using a load balancer IP that is routable from your VPN connection into the private network]] !!!!.6. Run Kubernetes API requests against your cluster by using the IAM ID token that you retrieved earlier. For example, list the Kubernetes version that runs in your cluster.
If you enabled SSL certificate verification in your API test framework, make sure to disable this feature.

GET <masterURL>/version

Header
: <-- <ul> -->

* Authorization: bearer <id_token>

<-- </ul> -->

Path
: <-- <ul> -->

* <masterURL>: The service endpoint of your Kubernetes master that you retrieved in the previous step.

<-- </ul> -->

The following example shows the output of the previous API request.

{
"major": "1",
"minor": "1.26",
"gitVersion": "v1.26+IKS",
"gitCommit": "c35166bd86eaa91d17af1c08289ffeab3e71e11e",
"gitTreeState": "clean",
"buildDate": "2019-03-21T10:08:03Z",
"goVersion": "go1.11.5",
"compiler": "gc",
"platform": "linux/amd64"
}
7. Review the Kubernetes API documentation]] ! ! ! ! to find a list of supported APIs for the latest Kubernetes version.
service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: private
namespace: default
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 8080 Or, the <private_service_endpoint_port> that you found earlier.
targetPort: 8080 Optional. By default, the targetPort is set to match the port value unless specified otherwise.
---
kind: Endpoints
apiVersion: v1
metadata:
name: kube-api-via-nlb
subsets:
- addresses:
- ip: 172.20.0.1
ports:
- port: 2040
Show more
3. To create the private NLB, you must be connected to the cluster master. Because you can't yet connect through the private cloud service endpoint from a VPN or IBM Cloud Direct Link, you must connect to the cluster master and create the NLB by using the public cloud service endpoint or Kubernetes dashboard.



* If you enabled the private cloud service endpoint only, you can use the Kubernetes dashboard to create the NLB. The dashboard automatically routes all requests to the private cloud service endpoint of the master.



1. Log in to the [IBM Cloud console](https://cloud.ibm.com/).
2. From the menu bar, select the account that you want to use.
3. From the menu ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/icons/icon_hamburger.svg), click Kubernetes.
4. On the Clusters page, click the cluster that you want to access.
5. From the cluster detail page, click the Kubernetes Dashboard.
6. Click + Create.
7.
You can avoid subnet conflicts when you [create a cluster from the CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_cluster_create) by specifying a custom subnet CIDR for pods in the --pod-subnet option and a custom subnet CIDR for services in the --service-subnet option.
* If your VPN solution preserves the source IP addresses of requests, you can [create custom static routes](https://cloud.ibm.com/docs/openshift?topic=openshift-static-routes) to ensure that your worker nodes can route responses from your cluster back to your on-premises network.
* Note that the 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16 subnet ranges are prohibited because they are reserved for Red Hat OpenShift on IBM Cloud control plane functionality.









 External communication to apps that run on worker nodes 

Allow public or private traffic requests from outside the cluster to your apps that run on worker nodes.



 Private traffic to cluster apps 

When you deploy an app in your cluster, you might want to make the app accessible to only users and services that are on the same private network as your cluster. Private load balancing is ideal for making your app available to requests from outside the cluster without exposing the app to the general public. You can also use private load balancing to test access, request routing, and other configurations for your app before you later expose your app to the public with public network services. To allow private traffic requests from outside the cluster to your apps, you can create private Kubernetes networking services, such as private NodePorts, NLBs, and Ingress ALBs. You can then use Calico pre-DNAT policies to block traffic to public NodePorts of private networking services. For more information, see [Planning private external load balancing](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_network_planningprivate_access).





 Public traffic to cluster apps
Use the IBM Cloud Kubernetes Service observability plug-in to create a monitoring configuration for IBM Cloud Monitoring in your cluster, and use this monitoring configuration to automatically collect and forward metrics to IBM Cloud Monitoring.

You can set up a monitoring configuration for your cluster from the console or CLI.



* [Forward metrics to an IBM Cloud Monitoring instance](https://cloud.ibm.com/docs/containers?topic=containers-health-monitormonitoring)







 Forward Kubernetes API audit logs to Log Analysis 

To forward audit logs to IBM Cloud Log Analysis, you can [create a Kubernetes audit system by using the provided image and deployment](https://cloud.ibm.com/docs/containers?topic=containers-health-auditaudit-api-server-la).





 Next Steps 

The next step on the deployment journey is:



* [Configure DNS](https://cloud.ibm.com/docs/cloud-native-journey?topic=cloud-native-journey-cloud-native-configure-dns)
If both the public and private cloud service endpoints are enabled for your cluster, the public cloud service endpoint is listed by default for the masterURL. To use the private cloud service endpoint instead, find the URL in the privateServiceEndpointURL field of the output.

GET https://containers.cloud.ibm.com/global/v2/getCluster?cluster=<cluster_name_or_ID>

Header
: <-- <ul> -->

* Authorization: Your IBM Cloud IAM access token.

<-- </ul> -->

Path
: <-- <ul> -->

* <cluster_name_or_ID>: The name or ID of your cluster that you retrieved with the GET https://containers.cloud.ibm.com/global/v2/classic/getClusters or GET https://containers.cloud.ibm.com/global/v2/vpc/getClusters?provider=vpc-gen2 API in Automating cluster deployments with the API]]!.

<-- </ul> -->

The following example shows output for a public cloud service endpoint request.

...
"etcdPort": "31593",
"masterURL": "https://c2.us-south.containers.cloud.ibm.com:30422",
"ingress": {
...}

The following example shows output for a private cloud service endpoint request.

...
"etcdPort": "31593",
"masterURL": "https://c2.private.us-south.containers.cloud.ibm.com:30422",
"ingress": {
...}
5. To use a private cloud service endpoint, you must first expose the private cloud service endpoint by using a load balancer IP that is routable from your VPN connection into the private network]] !!!!.6. Run Kubernetes API requests against your cluster by using the IAM ID token that you retrieved earlier.
For more information about the service, see the [Activity Tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started) documentation. For more information about the types of events that you can track, see [Activity Tracker events](https://cloud.ibm.com/docs/containers?topic=containers-at_events). Yes Yes 
 IBM Log Analysis Add log management capabilities to your cluster by deploying a Log Analysis agent to your worker nodes to manage logs from your pod containers. For more information, see [Managing Kubernetes cluster logs with IBM Log Analysis](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-tutorial-use-logdna). Yes Yes 
 IBM Cloud Monitoring Gain operational visibility into the performance and health of your apps by deploying a Monitoring agent to your worker nodes to forward metrics to IBM Cloud® Monitoring. For more information, see [Analyzing metrics for an app that is deployed in a Kubernetes cluster](https://cloud.ibm.com/docs/monitoring?topic=monitoring-kubernetes_clusterkubernetes_cluster). Yes Yes 
 Instana Instana provides infrastructure and app performance monitoring with a GUI that automatically discovers and maps your apps. Instana captures every request to your apps, which you can use to troubleshoot and perform root cause analysis to prevent the problems from happening again. For more information, see the [Instana product page](https://www.ibm.com/products/instana) Yes Yes 
 Prometheus Prometheus is an open source monitoring, logging, and alerting tool that was designed for Kubernetes. Prometheus retrieves detailed information about the cluster, worker nodes, and deployment health based on Kubernetes logging information. CPU, memory, I/O, and network activity is collected for each container that runs in a cluster. You can use the collected data in custom queries or alerts to monitor performance and workloads in your cluster.
You can avoid subnet conflicts when you [create a cluster from the CLI](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_cluster_create) by specifying a custom subnet CIDR for pods in the --pod-subnet option and a custom subnet CIDR for services in the --service-subnet option.
* If your VPN solution preserves the source IP addresses of requests, you can [create custom static routes](https://cloud.ibm.com/docs/containers?topic=containers-static-routes) to ensure that your worker nodes can route responses from your cluster back to your on-premises network.
* Note that the 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16 subnet ranges are prohibited because they are reserved for IBM Cloud Kubernetes Service control plane functionality.









 External communication to apps that run on worker nodes 

Allow public or private traffic requests from outside the cluster to your apps that run on worker nodes.



 Private traffic to cluster apps 

When you deploy an app in your cluster, you might want to make the app accessible to only users and services that are on the same private network as your cluster. Private load balancing is ideal for making your app available to requests from outside the cluster without exposing the app to the general public. You can also use private load balancing to test access, request routing, and other configurations for your app before you later expose your app to the public with public network services. To allow private traffic requests from outside the cluster to your apps, you can create private Kubernetes networking services, such as private NodePorts, NLBs, and Ingress ALBs. You can then use Calico pre-DNAT policies to block traffic to public NodePorts of private networking services. For more information, see [Planning private external load balancing](https://cloud.ibm.com/docs/containers?topic=containers-cs_network_planningprivate_access).





 Public traffic to cluster apps