vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Optional: Customize the public Ingress controller's routing rules with [optional configurations](https://docs.openshift.com/container-platform/4.11/networking/routes/route-configuration.html). For example, you can use [route-specific HAProxy annotations](https://docs.openshift.com/container-platform/4.11/networking/routes/route-configuration.htmlnw-route-specific-annotations_route-configuration).
11. To create routes for more apps by using the same subdomain, you can repeat steps 7 - 10 so that the route is generated by the same public Ingress controller. If you want to create routes for more apps by using a different subdomain, repeat all steps in this section to create a new public Ingress controller with a different domain.









 Setting up private routes 

Use a private Ingress controller to expose apps in your cluster on the private network.

The method for setting up private routes varies depending on your cluster's infrastructure provider and your service endpoint setup.



* [Setting up private routes in classic clusters or in VPC clusters with a public cloud service endpoint](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_routesprivate-routes-setup-43)
* [Setting up private routes in VPC clusters with a private cloud service endpoint only](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_routesroutes-private-vpc-privse)





 Setting up private routes in classic clusters or in VPC clusters with a public cloud service endpoint 

If your cluster is created on classic infrastructure, or if your cluster is created on VPC infrastructure and you enabled the public cloud service endpoint during cluster creation, your cluster is created with only a public Ingress controller by default. To privately expose your apps, you must first create a private IngressController resource and configure the controller with a subdomain. The Ingress operator automatically creates and configures a new private Ingress controller, which you can use to create private routes for your apps.
RPO (Recovery Point Objective) and RTO (Recovery Time Objective) for this configuration is less than 60 seconds.
* [Asynchronous DR](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/2-asynchronous-dr-nodes-are-across-different-regions-datacenters): Your Kubernetes clusters are deployed in different regions, such as us-south and us-east. Each cluster has its own Portworx installation and uses a separate Portworx key-value store that is not shared. To replicate data between clusters, you must set up scheduled replication between these clusters. Because of the higher latency and scheduled replication times, the RPO for this scenario might be up to 15 minutes.



To include your cluster in a Portworx disaster recovery configuration:



1. [Choose the disaster recovery configuration that works for your cluster setup](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/).
2. Review the prerequisites for the [Metro DR](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/px-metro/1-install-px/prerequisites) and [Asynchronous DR](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/async-dr/pre-requisites) configuration.
3. Configure disaster recovery for your cluster. Metro DR:



1. Choose at least two Kubernetes clusters that are located in the same metro location. If you have one cluster only, you can still configure this cluster for metro disaster recovery, but Portworx can't do a proper failover until a second cluster is configured.
2.
* [Creating a secret to store the KMS credentials](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_planpx_create_km_secret)



[Preparing your cluster for Portworx](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_preparingstorage_portworx_preparing)



* [Classic clusters](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_preparingpx-create-classic-volumes)
* [VPC clusters](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_preparingpx-create-vpc-volumes)
* [Satellite clusters](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_preparingpx-create-satellite-volumes)



[Setting up the Portworx key-value store](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_storestorage_portworx_kv_store)



* [Setting up a key-value store for Portworx metadata](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_storeportworx_database)
* [Using the Portworx key-value database](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_storeportworx-kvdb)



* [Optional: Setting up a Databases for etcd service instance](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_storeportworx-kv-db)





[Understanding encryption for Portworx](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_encryptionstorage_portworx_encryption)
For more information, see [Setting up a continuous delivery pipeline for a cluster](https://cloud.ibm.com/docs/containers?topic=containers-cicdcontinuous-delivery-pipeline).

Security for VPC clusters
: Added information for how to achieve [network segmentation and privacy in VPC clusters](https://cloud.ibm.com/docs/containers?topic=containers-securitynetwork_segmentation_vpc).





 1 October 2019 

End of service of IBM Log Analysis and IBM Cloud Monitoring
: Removed steps for using IBM Log Analysis and IBM Cloud Monitoring to work with cluster logs and metrics. You can collect logs and metrics for your cluster by setting up [IBM Log Analysis](https://cloud.ibm.com/docs/containers?topic=containers-healthlogging) and [IBM Cloud Monitoring](https://cloud.ibm.com/docs/monitoring?topic=monitoring-kubernetes_clusterkubernetes_cluster) instead.

New! Gateway-enabled classic clusters
: Keep your compute workloads private and allow limited public connectivity to your classic cluster by enabling a gateway. You can enable a gateway only on standard, Kubernetes clusters during cluster creation.
: When you enable a gateway on a classic cluster, the cluster is created with a compute worker pool of compute worker nodes that are connected to a private VLAN only, and a gateway worker pool of gateway worker nodes that are connected to public and private VLANs. Traffic into or out of the cluster is routed through the gateway worker nodes, which provide your cluster with limited public access. For more information, check out the following links. - Isolating networking workloads to edge nodes in classic gateway-enabled clusters - Flow of traffic to apps when using an [NLB 1.0](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancer-about), an [NLB 2.0](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancer-about), or Ingress ALBs.
: Ready to get started?
Preparing your cluster for Portworx 

If you want to build your Portworx storage layer on non-SDS worker nodes in your classic cluster must add raw, unformatted, and unmounted block storage to your worker nodes first. For VPC clusters, you can either attach storage before installing Portworx or allow Portworx to dynamically create Cloud Drives by using the Block Storage for VPC driver during installation.

Raw block storage can't be provisioned by using Kubernetes persistent volume claims (PVCs) as the block storage device is automatically formatted by IBM Cloud Kubernetes Service. Instead, you can use the IBM Cloud Block Volume Attacher plug-in in classic clusters or the VPC console, CLI, or API in VPC clusters to add block storage to your worker nodes.

Portworx supports block storage only. Worker nodes that mount file or object storage can't be used for the Portworx storage layer.

Keep in mind that the networking of non-SDS worker nodes in classic clusters is not optimized for Portworx and might not offer the performance benefits that your app requires.



 Classic clusters 

Classic



1. [Install the IBM Cloud Block Volume Attacher plug-in](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesblock_storage_attacher).
2. [Manually add block storage](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesmanual_block) to your worker nodes. For highly available data storage, Portworx requires at least 3 worker nodes with raw and unformatted block storage.
3. If you want to use [journal devices](https://docs.portworx.com/install-with-other/operate-and-maintain/performance-and-tuning/tuning/), choose from the following options. Journal devices allow Portworx to write logs directly to a local disk on your worker node.



* Attach an additional disk to at least 3 worker nodes to use for the journal.
RPO (Recovery Point Objective) and RTO (Recovery Time Objective) for this configuration is less than 60 seconds.
* [Asynchronous DR](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/2-asynchronous-dr-nodes-are-across-different-regions-datacenters): Your Red Hat OpenShift clusters are deployed in different regions, such as us-south and us-east. Each cluster has its own Portworx installation and uses a separate Portworx key-value store that is not shared. To replicate data between clusters, you must set up scheduled replication between these clusters. Because of the higher latency and scheduled replication times, the RPO for this scenario might be up to 15 minutes.



To include your cluster in a Portworx disaster recovery configuration:



1. [Choose the disaster recovery configuration that works for your cluster setup](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/).
2. Review the prerequisites for the [Metro DR](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/px-metro/1-install-px/prerequisites) and [Asynchronous DR](https://docs.portworx.com/portworx-install-with-kubernetes/disaster-recovery/async-dr/pre-requisites) configuration.
3. Configure disaster recovery for your cluster. Metro DR:



1. Choose at least two Red Hat OpenShift clusters that are located in the same metro location. If you have one cluster only, you can still configure this cluster for metro disaster recovery, but Portworx can't do a proper failover until a second cluster is configured.
2.
[Set up sidecar injection for your app microservices, deploy the app microservices into a namespace, and create Kubernetes services for the app microservices so that they are in the Istio service mesh](https://cloud.ibm.com/docs/containers?topic=containers-istio-meshistio_sidecar).



To publicly expose apps:



1. Register the load balancer IP or hostname by creating a DNS subdomain. For more information about registering DNS subdomains in IBM Cloud Kubernetes Service, see [Classic: Registering an NLB subdomain](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancer_hostname) or [Registering a VPC load balancer hostname with a DNS subdomain](https://cloud.ibm.com/docs/containers?topic=containers-vpc-lbaasvpc_lb_dns).



* Classic clusters:

ibmcloud ks nlb-dns create classic --cluster <cluster_name_or_id> --ip <LB_IP> [--ip <LB_zone2_IP> ...]
* VPC clusters:

ibmcloud ks nlb-dns create vpc-gen2 -c <cluster_name_or_ID> --lb-host <LB_hostname>



2. Verify that the subdomain is created. In the output, copy the name of your SSL secret in the SSL Cert Secret Name field.

ibmcloud ks nlb-dns ls --cluster <cluster_name_or_id>

 Example output for classic clusters:
Hostname IP(s) Health Monitor SSL Cert Status SSL Cert Secret Name
mycluster-a1b2cdef345678g9hi012j3kl4567890-0001.us-south.containers.appdomain.cloud ["168.1.1.1"] None created <certificate>

 Example output for VPC clusters:
Preparing your cluster for Portworx 

If you want to build your Portworx storage layer on non-SDS worker nodes in your classic cluster must add raw, unformatted, and unmounted block storage to your worker nodes first. For VPC clusters, you can either attach storage before installing Portworx or allow Portworx to dynamically create Cloud Drives by using the Block Storage for VPC driver during installation.

Raw block storage can't be provisioned by using Kubernetes persistent volume claims (PVCs) as the block storage device is automatically formatted by Red Hat OpenShift on IBM Cloud. Instead, you can use the IBM Cloud Block Volume Attacher plug-in in classic clusters or the VPC console, CLI, or API in VPC clusters to add block storage to your worker nodes.

Portworx supports block storage only. Worker nodes that mount file or object storage can't be used for the Portworx storage layer.

Keep in mind that the networking of non-SDS worker nodes in classic clusters is not optimized for Portworx and might not offer the performance benefits that your app requires.



 Classic clusters 

Classic



1. [Install the IBM Cloud Block Volume Attacher plug-in](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesblock_storage_attacher).
2. [Manually add block storage](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesmanual_block) to your worker nodes. For highly available data storage, Portworx requires at least 3 worker nodes with raw and unformatted block storage.
3. If you want to use [journal devices](https://docs.portworx.com/install-with-other/operate-and-maintain/performance-and-tuning/tuning/), choose from the following options. Journal devices allow Portworx to write logs directly to a local disk on your worker node.



* Attach an additional disk to at least 3 worker nodes to use for the journal.
Right-click the vDS and click Edit Settings.
2. On the Properties tab, select the Advanced option.
3. Ensure that the Maximum MTU value is set to 1500.

When the MTU size in a vDS is changed, the attached uplinks (physical NICs) are brought down and up again. As a result, a brief outage occurs for the VMs that are using the uplink. Therefore, plan the MTU setting update during scheduled downtime.









 Distributed switch allocation 

The allocation of distributed switches varies if you have existing instances and clusters. Review the following considerations for switch creation when you create a cluster:



* If one or more existing clusters are in the same pod that uses distributed switches that are named SDDC-DSwitch-Private and SDDC-DSwitch-Public, your new cluster uses the same switches as the existing cluster.
* If one or more existing clusters are in the same pod, and the pod uses distributed switches that are named by using the same name as the pod (rather than named by using the same name as the cluster), your new cluster uses the same switches as the existing cluster.
* If no existing cluster is in the same pod, or all clusters in that pod distribute switches that are named by using the same name as the cluster rather than the pod, then your new cluster is configured with the new switch whose name is based only on the pod.
* For vSphere 7, each cluster has its own-distributed switch pair that is named <instance_name>-<cluster_name>-public and <instance_name>-<cluster_name>-private.







 EVC mode settings 

Review the following table for an overview of the EVC (Enhanced VMware vMotion Compatibility) mode settings for vCenter Server instances and the differences between vSphere versions.



Table 7. EVC mode settings for vCenter Server instances and clusters

 Bare metal server CPU model vSphere 6.7<br><br>[10] vSphere 7.0 

 Skylake EVC is set to Intel® Broadwell Generation. Skylake is not supported. 
 Cascade Lake For the management cluster, EVC is not set. For all other clusters, EVC is set to Intel Skylake Generation.
Furthermore, if you need to use the ibmcloud oc cluster service bind command to [integrate with an IBM Cloud service](https://cloud.ibm.com/docs/openshift?topic=openshift-service-bindingbind-services), that service must be in the same resource group as the cluster. Services that don't use resource groups like IBM Cloud Container Registry or that don't need service binding like IBM Log Analysis work even if the cluster is in a different resource group.
: 

* Consider giving clusters unique names across resource groups and regions in your account to avoid naming conflicts. You can't rename a cluster.
* You can assign users an access role to a resource group to grant permissions as described in the following scenarios. Note that unlike resource instances, you can't grant access to an individual instance within a resource group.
* All IBM Cloud IAM services in the resource group, including all clusters in Red Hat OpenShift on IBM Cloud and images in IBM Cloud Container Registry.
* All instances within a service in the resource group, such as all the clusters in Red Hat OpenShift on IBM Cloud.
* All instances within a region of a service in the resource group, such as all the clusters in the US South region of Red Hat OpenShift on IBM Cloud.
: For classic clusters in Dallas, updated the required IP addresses and ports that you must open in a [public gateway firewall device](https://cloud.ibm.com/docs/containers?topic=containers-firewallfirewall_outbound), [private gateway device firewall](https://cloud.ibm.com/docs/containers?topic=containers-firewallfirewall_private), or [Calico network isolation policies](https://github.com/IBM-Cloud/kube-samples/tree/master/calico-policies).

Ingress classes
: Added information about [specifying Ingress classes](https://cloud.ibm.com/docs/containers?topic=containers-managed-ingress-aboutmanaged-ingress-class) to apply Ingress resources to specific ALBs.

Object Storage
: Added steps to help you [decide on the object storage configuration](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_installconfigure_cos) and added troubleshooting steps for when [app pods fail because of an Operation not permitted error](https://cloud.ibm.com/docs/containers?topic=containers-cos_operation_not_permitted).





 13 October 2020 

New! Certified Kubernetes version 1.19
: You can now create clusters that run Kubernetes version 1.19. To update an existing cluster, see the [Version 1.19 preparation actions](https://cloud.ibm.com/docs/containers?topic=containers-cs_versions_119). The Kubernetes 1.19 release is also certified.

Deprecated: Kubernetes version 1.16
: With the release of version 1.19, clusters that run version 1.16 are deprecated. Update your clusters to at least [version 1.17](https://cloud.ibm.com/docs/containers?topic=containers-cs_versionsk8s_version_archive) today.

New! Network load balancer for VPC
ibmcloud sat endpoint get --location <location_ID> --endpoint <endpoint_ID> [> endpoint1.txt]
6. List the clusters that are created in your Satellite location.

ibmcloud oc cluster ls --provider satellite
7. Re-create your location.



* For options to create the location and hosts, see [Deciding how to create your Satellite location](https://cloud.ibm.com/docs/satellite?topic=satellite-locations).
* To create Satellite Link endpoints, see [cloud](https://cloud.ibm.com/docs/satellite?topic=satellite-link-cloud-createlink-cloud) or [location](https://cloud.ibm.com/docs/satellite?topic=satellite-link-cloud-createlink-location) endpoint instructions.
* For clusters, see [Creating Red Hat OpenShift in Satellite](https://cloud.ibm.com/docs/openshift?topic=openshift-satellite-clusters). If you manually deployed Kubernetes configurations instead of using Satellite Config, also see [Copying deployments to another cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-update_appcopy_apps_cluster).



8. Redeploy your apps to your cluster. Because Satellite Config works across Satellite locations, you can [add your new clusters to existing cluster groups](https://cloud.ibm.com/docs/satellite?topic=satellite-setup-clusters-satconfig-groups) that are already [subscribed to the configurations](https://cloud.ibm.com/docs/satellite?topic=satellite-satcon-manage-direct-upload) that you want to deploy.
9. Similar to Satellite Config, if you used Satellite storage configurations, you can assign these configurations to your new clusters to install the storage drivers.
* [Setting up public routes in classic clusters or in VPC clusters with a public cloud service endpoint](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_routesroutes-public-classic)
* [Setting up public routes in VPC clusters with a private cloud service endpoint only](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_routesroutes-public-vpc-privse)





 Setting up public routes in classic clusters or in VPC clusters with a public cloud service endpoint 

If your cluster is created on classic infrastructure, or if your cluster is created on VPC infrastructure and you enabled the public cloud service endpoint during cluster creation, your cluster is created with a public Ingress controller by default. You can use this Ingress controller to create public routes for your app.



1. Create a Kubernetes ClusterIP service for your app deployment. The service provides an internal IP address for the app that the Ingress controller can send traffic to.

oc expose deploy <app_deployment_name> --name my-app-svc
2. Choose a domain for your app. Note that route URLs must be 130 characters or fewer IBM-provided domain: If you don't need to use a custom domain, a route hostname is generated for you in the format <service_name>-<project>.<cluster_name>-<random_hash>-0000.<region>.containers.appdomain.cloud. Custom domain: To specify a custom domain, work with your DNS provider or [IBM Cloud® Internet Services](https://cloud.ibm.com/catalog/services/internet-services).



1. Get the public IP address for the public Ingress controller service in each zone in the EXTERNAL-IP column. Note that the Ingress controller service in the first zone where you have workers nodes is always named router-default, and Ingress controller services in zones that you subsequently add to your cluster have names such as router-dal12.

oc get svc -n openshift-ingress
2.
istio-ingressgateway LoadBalancer 172.21.XXX.XXX 169.1.1.1 80:31380/TCP,443:31390/TCP,31400:31400/TCP,5011:31323/TCP,8060:32483/TCP,853:32628/TCP,15030:31601/TCP,15031:31915/TCP 22m

 Example output for VPC clusters:
istio-ingressgateway LoadBalancer 172.21.XXX.XXX 1234abcd-us-south.lb.appdomain.cloud 80:31380/TCP,443:31390/TCP,31400:31400/TCP,5011:31323/TCP,8060:32483/TCP,853:32628/TCP,15030:31601/TCP,15031:31915/TCP 22m
6. Register the load balancer IP or hostname by creating a DNS subdomain. For more information about registering DNS subdomains in IBM Cloud Kubernetes Service, see [Classic: Registering an NLB subdomain](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancer_hostname) or [Registering a VPC load balancer hostname with a DNS subdomain](https://cloud.ibm.com/docs/containers?topic=containers-vpc-lbaasvpc_lb_dns). Example command for Classic clusters.

ibmcloud ks nlb-dns create classic --cluster <cluster_name_or_id> --ip <LB_IP> [--ip <LB_zone2_IP> ...]

Example command for VPC clusters.

ibmcloud ks nlb-dns create vpc-gen2 -c <cluster_name_or_ID> --lb-host <LB_hostname>
7. Verify that the subdomain is created. In the output, copy the name of your SSL secret in the SSL Cert Secret Name field.