vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Changing service endpoints or VLAN connections 

After you initially set up your network when you [create a cluster](https://cloud.ibm.com/docs/containers?topic=containers-clusters), you can change the service endpoints that your cluster master is accessible through or change the VLAN connections for your worker nodes.

The content on this page is specific to classic clusters only. For information about VPC clusters, see [Understanding network basics of VPC clusters](https://cloud.ibm.com/docs/containers?topic=containers-vpc-subnets).



 Setting up the private cloud service endpoint 

Enable the private cloud service endpoint for your cluster.

The private cloud service endpoint makes your Kubernetes master privately accessible. Your worker nodes and your authorized cluster users can communicate with the Kubernetes master over the private network. To determine whether you can enable the private cloud service endpoint, see [Worker-to-master and user-to-master communication](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsworkeruser-master). Note that you can't disable the private cloud service endpoint after you enable it.

Did you create a cluster with only a private cloud service endpoint before you enabled your account for [VRF](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointvrf) and [service endpoints](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointservice-endpoint)? Try [setting up the public cloud service endpoint](https://cloud.ibm.com/docs/containers?topic=containers-cs_network_clusterset-up-public-se) so that you can use your cluster until your support cases are processed to update your account.



1. Enable [VRF](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointvrf) in your IBM Cloud infrastructure account. To check whether a VRF is already enabled, use the ibmcloud account show command.
2.
If, for example, the IP address is from an on-premises network and the control plane can't reach the IP address, the webhook service does not work.
* If the URL is a cluster IP address, which means the webhook service is inside of the cluster, the Kubernetes API needs to connect to cluster network. If you have cluster version 1.21 and later, and your webhook uses the cluster IP address, you must update your webhook to use a Kubernetes service instead.









 I need help with a broken webhook. What can I do? 

For help troubleshooting webhooks, see [Debugging webhooks](https://cloud.ibm.com/docs/containers?topic=containers-ts-webhook-debug) or [Cluster can't update because of broken webhook](https://cloud.ibm.com/docs/containers?topic=containers-webhooks_update).
By issuing the update command, the worker nodes are reloaded to pick up the service endpoint configuration. If no worker update is available, you must reload the worker nodes manually with the ibmcloud ks worker reload[command](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_worker_reload). If you reload, be sure to cordon, drain, and manage the order to control the maximum number of worker nodes that are unavailable at a time.









 Switching from the public cloud service endpoint to the private cloud service endpoint 

Enable worker nodes to communicate with the master over the private network instead of the public network by enabling the private cloud service endpoint.

All clusters that are connected to a public and a private VLAN use the public cloud service endpoint by default. Your worker nodes and your authorized cluster users can securely communicate with the Kubernetes master over the public network. To enable worker nodes to communicate with the Kubernetes master over the private network instead of the public network, you can enable the private cloud service endpoint. Then, you can optionally disable the public cloud service endpoint.



* If you enable the private cloud service endpoint and keep the public cloud service endpoint enabled too, workers always communicate with the master over the private network, but your users can communicate with the master over either the public or private network.
* If you enable the private cloud service endpoint but disable the public cloud service endpoint, workers and users must communicate with the master over the private network.



Note that you can't disable the private cloud service endpoint after you enable it.



1. Enable [VRF](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointvrf) in your IBM Cloud infrastructure account. To check whether a VRF is already enabled, use the ibmcloud account show command.
2. [Enable your IBM Cloud account to use service endpoints](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointservice-endpoint).
3. Enable the private cloud service endpoint.
: Worker node fix pack update change log documentation is available for version [4.6.22_1536_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_464622_1536), [4.5.35_1534_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4535_1534), [4.4.33_1537_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4433_1537), and [3.11.404_1584_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_311311404_1584).





 25 March 2021 

CIS Kubernetes Benchmark
: Added information about Red Hat OpenShift on IBM Cloud compliance with the [Center for Internet Security (CIS) Kubernetes Benchmark](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark) for clusters that run Red Hat OpenShift version 4.5 or later.

CRI-O container runtime
: Added [information about the CRI-O container runtime interface](https://cloud.ibm.com/docs/openshift?topic=openshift-securitycontainer-runtime) for worker nodes.

Certificate Manager instances
: The default Certificate Manager instance for your cluster is now named in the format kube-crtmgr-<cluster_ID>.





 22 March 2021 

Ingress updates
: Updated the [nginx-ingress build to 2466 and the ingress-auth build to 982](https://cloud.ibm.com/docs/containers?topic=containers-cluster-add-ons-changelogkube_ingress_changelog) for the Red Hat OpenShift on IBM Cloud Ingress image.





 17 March 2021
After upgrading my classic cluster to version 1.21, I'm finding connectivity issues 

There is a known issue with classic Kubernetes 1.21 clusters that have public and private service endpoint, but don't have VRF enabled.

 What’s happening 

You upgraded your classic cluster to Kubernetes 1.21 and are finding connectivity issues such as connecting to the Kubernetes console, fetching logs from pods, or running the IBM Cloud Kubernetes Service Diagnostics and Debug Tool.

 Why it’s happening 

In Kubernetes version 1.21, Konnectivity replaces OpenVPN as the network proxy that is used to secure the communication of the Kubernetes API server master to worker nodes in the cluster.

However, when using Konnectivity, a problem exists with masters to cluster nodes communication when all the following conditions are met.



* You are updating an existing classic cluster to Kubernetes version 1.21 or are deploying a new classic cluster.
* The cluster has both private and public service endpoints enabled.
* Service endpoints or VRF are not enabled in the account.



To determine if VRF and service endpoints are enabled in your account, run ibmcloud account show. Look for the following output.

VRF Enabled: true
Service Endpoint Enabled: true

To determine if your classic cluster has both public and private service endpoint enabled, run ibmcloud ks cluster get -c <cluster_id>. Look for output similar to:

Public Service Endpoint URL: https://c105.<REGION>.containers.cloud.ibm.com:<port>
Private Service Endpoint URL: https://c105.private.<REGION>.containers.cloud.ibm.com:<port>

 How to fix it 

If your cluster meets these conditions, delay updating to version 1.21 until you can [enable service endpoints and VRF](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint&interface=uivrf) for the account.
HPCS Router add-on
: Versions 4.6.0_1570, 4.7.0_1561, 4.8.0_1560, and 4.9.0_1559 of the [HPCS Router add-on](https://cloud.ibm.com/docs/openshift?topic=openshift-hpcs-router-changelog) are released. Cloud Image Key Synchronizer add-on
: Version 1.0.0_956 of the [Cloud Image Key Synchronizer add-on](https://cloud.ibm.com/docs/openshift?topic=openshift-image-key-synchronizer-changelog) is released.







 April 2022 



 28 April 2022 

CLI change log update
: The [CLI change log](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_cli_changelog) is updated for version 1.0.404.

Certified Kubernetes
: Version [4.10](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_410) release is now certified.

CIS Kubernetes Benchmark
: Added information about Red Hat OpenShift on IBM Cloud compliance with the [Center for Internet Security (CIS) Kubernetes Benchmark](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-410) for clusters that run Red Hat OpenShift version 4.10.

Cluster security groups
: You can now specify up to five security groups to attach to workers when you create a VPC cluster. For more information, see [Adding VPC security groups to clusters and worker pools during create time](https://cloud.ibm.com/docs/containers?topic=containers-vpc-security-group&interface=uivpc-sg-cluster-workers).





 27 April 2022 

New! Red Hat OpenShift on IBM Cloud 4.10
The curl command would look something like this: curl --header 'Host: hello-world-service.ibmom.com' my-us-cluster-e7f2ca73139645ddf61a8702003a483a-0000.us-south.containers.appdomain.cloud/hostname









 Step 2: And then to another location 

Repeat the steps from above for the London location with the following replacements:



* In [Create a Kubernetes cluster](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-multi-region-k8s-cismulti-region-k8s-cis-3) replace:



* the cluster name my-us-cluster with my-uk-cluster;
* the location from North America and Dallas with Europe and London.



* In the [Deploy the application to the Kubernetes cluster](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-multi-region-k8s-cismulti-region-k8s-cis-deploy_application) replace:



* Replace the MYCLUSTER= my-us-cluster with my-uk-cluster



* [Configure the Ingress for the DNS subdomain](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-multi-region-k8s-cismulti-region-k8s-cis-ingress)







 Step 3: Configure multi-location load-balancing 

Your application is now running in two clusters but it is missing one component for the users to access either clusters transparently from a single entry point.

In this section, you will configure IBM Cloud Internet Services (CIS) to distribute the load between the two clusters. CIS is a one stop-shop service providing Global Load Balancer (GLB), Caching, Web Application Firewall (WAF) and Page rule to secure your applications while ensuring the reliability and performance for your Cloud applications.

To configure a global load balancer, you will need:



* to point a custom domain to CIS name servers,
Knative add-on is unsupported
: The Knative add-on is automatically removed from your cluster. Instead, use the [Knative open source project](https://knative.dev/docs/install/) or [IBM Cloud® Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-getting-started), which includes Knative's open-source capabilities.

Worker node versions
: Worker node fix pack update change log documentation is available for version [1.19.7_1532](https://cloud.ibm.com/docs/containers?topic=containers-changelog_1191197_1532), [1.18.15_1538](https://cloud.ibm.com/docs/containers?topic=containers-118_changelog11815_1538), [1.17.17_1551](https://cloud.ibm.com/docs/containers?topic=containers-117_changelog11717_1551), and [1.16.15_1557](https://cloud.ibm.com/docs/containers?topic=containers-116_changelog11615_1557).





 14 January 2021 

Cluster autoscaler
: Cluster autoscaler add-on patch update 1.0.1_210 is released. For more information, see the [Cluster autoscaler add-on change log](https://cloud.ibm.com/docs/containers?topic=containers-ca_changelog).





 12 January 2021 

Ingress resources
: Added example Ingress resource definitions that are compatible with Kubernetes version 1.19.

Kubernetes benchmarks
: Added how to [run the CIS Kubernetes benchmark tests on your own worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmarkcis-worker-test).

Removal of data center support
: Updated the documentation to reflect that Melbourne (mel01) is no longer available as an option to create IBM Cloud resources in.





 7 January 2021
Server: https://127.0.0.1:2040/api/v1/namespaces/ibm-kube-audit/services/ibmcloud-kube-audit-service/proxy/post
Policy: default
12. Apply the webhook to your Kubernetes API server by refreshing the cluster master. It might take several minutes for the master to refresh.

ibmcloud ks cluster master refresh --cluster <cluster_name_or_ID>
13. While the master refreshes, [provision an instance of IBM Log Analysis and deploy a logging agent to every worker node in your cluster](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-tutorial-use-logdna). The logging agent is required to forward logs from inside your cluster to the IBM Log Analysis service. If you already set up logging agents in your cluster, you can skip this step.
14. After the master refresh completes and the logging agents are running on your worker nodes, you can [view your Kubernetes API audit logs in IBM Log Analysis](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-tutorial-use-logdna).



After you set up the audit webhook in your cluster, you can monitor version updates to the kube-audit-to-logdna image by running ibmcloud cr image-list --include-ibm | grep ibmcloud-kube-audit. To see the version of the image that currently runs in your cluster, run kubectl get pods | grep ibmcloud-kube-audit to find the audit pod name, and run kubectl describe pod <pod_name> to see the image version.





 Forwarding Kubernetes API audit logs to a resource in the IBM Cloud private network 

Forward audit logs to a resource other than Log Analysis that is outside of your cluster and accessible in the IBM Cloud private network.

The following example uses the haproxytech/haproxy-alpine:2.6 image to forward logs. Do not use this image in production environments.
[Menu icon](https://cloud.ibm.com/docs-content/v1/content/4d497aff9e27b92fbbc2ee4e343a9ec4549cb1d7/icons/icon_hamburger.svg), click Kubernetes.
4. On the Clusters page, click the cluster that you want to access.
5. From the cluster detail page, click the Kubernetes Dashboard.
6. Click + Create.
7. Select Create from file, upload the kube-api-via-nlb.yaml file, and click Upload.
8. In the Overview page, verify that the kube-api-via-nlb service is created. In the External endpoints column, note the 10.x.x.x address. This IP address exposes the private cloud service endpoint for the Kubernetes master on the port that you specified in your YAML file.



* If you also enabled the public cloud service endpoint, you already have access to the master.



1. Download and add the kubeconfig configuration file for your cluster to your existing kubeconfig in /.kube/config or the last file in the KUBECONFIG environment variable.

ibmcloud ks cluster config -c <cluster_name_or_ID>
2. Create the NLB and endpoint.

kubectl apply -f kube-api-via-nlb.yaml
3. Verify that the kube-api-via-nlb NLB is created. In the output, note the 10.x.x.xEXTERNAL-IP address. This IP address exposes the private cloud service endpoint for the Kubernetes master on the port that you specified in your YAML file.

kubectl get svc -o wide

In this example output, the IP address for the private cloud service endpoint of the Kubernetes master is 10.186.92.42.

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
Before you create a cluster, you must enable your account to use service endpoints. To enable service endpoints, run ibmcloud account update --service-endpoint-enable true.

In VPC clusters in Red Hat OpenShift on IBM Cloud, you can't disable the private cloud service endpoint or set up a cluster with the public cloud service endpoint only.

Your VPC cluster is created with both a public and a private cloud service endpoint by default. To create clusters with worker nodes that are connected to the private network only, you must enable only the private service endpoint during cluster creation. Do not enable the public service endpoint. For example, to create a VPC cluster with only a private cloud service endpoint [in the CLI](https://cloud.ibm.com/docs/containers?topic=containers-cluster-create-vpc-gen2&interface=cli), include the --disable-public-service-endpoint option. If you include this option, your cluster is created with routers and Ingress controllers that expose your apps on the private network only by default. If you later want to expose apps to a public network, you must manually create public routers and Ingress controllers.





 Worker-to-master communication in VPC clusters 

Worker node communication to the Kubernetes master is established differently based on your cluster version.



* Worker node communication to the Kubernetes master is established over the [VPC virtual private endpoint (VPE)](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-subnetsvpc_basics_vpe). If the public cloud service endpoint is also enabled, worker-to-master traffic is established half over the public endpoint and half over the VPE for protection from potential outages of the public or private network.



To secure communication over public and private cloud service endpoints or VPEs, Red Hat OpenShift on IBM Cloud automatically sets up a Konnectivity connection between the Kubernetes master and worker nodes when the cluster is created. Worker nodes securely talk to the master through TLS certificates, and the master talks to workers through the Konnectivity connection.
Table 2. Ports on IBM Cloud

 Port Number Description 

 CA ports 
 Fabric CA 7054 Port 7054 is exposed from the CA container. This is exposed all the way to the customer application via the service. 
 Peer ports 
 Peer gRPC API 7051 This is the port used by the clients to communicate with the peer. 
 Smart contract containers 7052 This is the port used by the smart contract containers to communicate with the peer. 
 Peer Operations 9443 This port is used to get to the health check endpoint and to the metrics endpoint of the peer. 
 gRPC Web proxy 7443 This port is used by the UI to communicate with the peer via gRPC web APIs. 
 Ordering service ports 
 Orderer gRPC API 7050 This is the port used by the clients to communicate with the orderer. 
 Orderer Operations 8443 This port is used to get to the health check endpoint and to the metrics endpoint of the Orderer. 
 gRPC Web proxy 7443 This port is used by the UI to communicate with the orderer via gRPC web APIs. 







 Cluster and Operating System security 



* Sensitive data: Cluster configuration data is stored in the etcd component of your Kubernetes master. Data in etcd is stored on the local disk of the Kubernetes master and is backed up to IBM Cloud Object Storage. Data is encrypted during transit to IBM Cloud Object Storage.





* UBI Linux: The Fabric Docker images use [Red Hat UBI images](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/indexhow_are_ubi_images_different/), which is a smaller, lighter, and more secure version of Linux.







 Keys and cluster access information 



Table 3. Types of keys and information used to access the customer Kubernetes cluster

 Type Description Storage Access 

 Customer Kubernetes cluster access information In order for the service components to access and manage the components that are deployed into this cluster, the service components generate a Kubernetes secret for cluster access. Each Kubernetes secret is tied to that specific customer cluster.
: Master fix pack update change log documentation is available for version [1.20.5_1538](https://cloud.ibm.com/docs/containers?topic=containers-changelog_1201206_1538) and [1.19.10_1545](https://cloud.ibm.com/docs/containers?topic=containers-changelog_11911910_1545).







 April 2021 



 30 April 2021 

CIS Kubernetes Benchmark
: Added information about Red Hat OpenShift on IBM Cloud compliance with the [Center for Internet Security (CIS) Kubernetes Benchmark](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark) for clusters that run Red Hat OpenShift version 4.6.





 28 April 2021 

Default version
: The default version for new clusters is now 4.6.

Image key synchronizer
: Updated the version change log for the [Image Key Synchronizer](https://cloud.ibm.com/docs/openshift?topic=openshift-image-key-synchronizer-changelog) add-on.





 27 April 2021 

Master versions
: Master fix pack update change log documentation is available for version [4.6.23_1540_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_464623_1540), [4.5.37_1536_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4537_1536), [4.4.33_1539_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4433_1539), and [3.11.420_1586_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_311311420_1586).
To create a standard object storage instance, run ibmcloud resource service-instance-create <name> cloud-object-storage standard global and note its ID.

--workers NUMBER_WORKERS_PER_ZONE
: Specify at least 2 worker nodes to include in the cluster. For more information, see [What is the smallest size cluster that I can make?](https://cloud.ibm.com/docs/openshift?topic=openshift-faqssmallest_cluster). Every worker node is assigned a unique worker node ID and domain name that must not be manually changed after the cluster is created. Changing the ID or domain name prevents the Kubernetes master from managing your cluster.

--disable-public-service-endpoint
: To ensure that worker nodes and authorized cluster users communicate with the master through the private cloud service endpoint only, include this option to create the cluster without the public cloud service endpoint. If you include this option, your cluster is created with routers and Ingress controllers that expose your apps on the private network only by default. If you later want to expose apps to a public network, you must manually create public routers and Ingress controllers.

--pod-subnet SUBNET
: In the first cluster that you create in a VPC, the default pod subnet is 172.17.0.0/18. In the second cluster that you create in that VPC, the default pod subnet is 172.17.64.0/18. In each subsequent cluster, the pod subnet range is the next available, non-overlapping /18 subnet. If you plan to connect your cluster to on-premises networks through IBM Cloud® Direct Link or a VPN service, you can avoid subnet conflicts by specifying a custom subnet CIDR that provides the private IP addresses for your pods.
: When you choose a subnet size, consider the size of the cluster that you plan to create and the number of worker nodes that you might add in the future. The subnet must have a CIDR of at least /23, which provides enough pod IPs for a maximum of four worker nodes in a cluster.