vania-janet/multiturn-rag-retrieval-data huggingface dataset:
(https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/roks_ov_vpc_priv.png)

Figure 1.Network setup for a VPC cluster that extends an on-prem data center



 Worker-to-worker communication 

To achieve this setup, you create VPC subnets in each zone where you want to deploy worker nodes. To run default Red Hat OpenShift components such as the web console or OperatorHub, public gateways are required for these subnets. Then, you create a VPC cluster that uses these VPC subnets.

Note that you might have subnet conflicts between the default ranges for workers nodes, pods, and services, and the subnets in your on-premises networks. When you create your VPC subnets, you can choose [custom address prefixes](https://cloud.ibm.com/docs/vpc?topic=vpc-configuring-address-prefixes) and then create your cluster by using these subnets. Additionally, you can specify a custom subnet CIDR for pods and services by using the --pod-subnet and --service-subnet options in the ibmcloud oc cluster create command when you create your cluster.





 Worker-to-master and user-to-master communication 

When you create the cluster, you enable private cloud service endpoint only to allow worker-to-master and user-to-master communication over the private network. Cluster users must either be in your VPC network or connect through a [VPC VPN connection](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-vpnaas).





 Worker communication to other services or networks 

To connect your cluster with your on-premises data center, you can set up the VPC VPN service. The IBM Cloud VPC VPN connects your entire VPC to an on-premises data center.
Instead, you can [remove the security group from the VPC ALB](https://cloud.ibm.com/docs/vpc-infrastructure-cli-plugin?topic=vpc-infrastructure-cli-plugin-vpc-reference&interface=clisecurity-group-target-remove) and [replace it with a security group](https://cloud.ibm.com/docs/vpc-infrastructure-cli-plugin?topic=vpc-infrastructure-cli-plugin-vpc-reference&interface=clisecurity-group-target-add) that you create and manage.





 Example scenarios for VPC cluster network setups 

Now that you understand the basics of cluster networking, check out some example scenarios in which various VPC cluster network setups can meet your workload needs.



 Scenario: Run internet-facing app workloads in a VPC cluster 

In this scenario, you run workloads in a VPC cluster that are accessible to requests from the Internet. Public access is controlled by security groups so that end users can access your apps while unwanted public requests to your apps are denied. Additionally, your workers have automatic access to any IBM Cloud services that support private cloud service endpoints.

Zoom

![Network setup for a VPC cluster that runs internet-facing app workloads.](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/roks_ov_vpc_pub.png)

Figure 1. Network setup for a VPC cluster that runs internet-facing app workloads





 Worker-to-worker communication 

To achieve this setup, you create VPC subnets in each zone where you want to deploy worker nodes. To run default Red Hat OpenShift components such as the web console or OperatorHub, public gateways are required for these subnets. Then, you create a VPC cluster that uses these VPC subnets.





 Worker-to-master and user-to-master communication
The source IP address of the request package is changed to the public IP address of the worker node where the router pod runs. Each router sends requests to the app instances in its own zone and to app instances in other zones. Additionally, if multiple app instances are deployed in one zone, the router alternates requests between app pods.
5. When the app returns a response packet, it uses the IP address of the worker node where the router that forwarded the client request exists. The router then sends the response packet through the load balancer service to the client.







 Traffic flow in a multizone VPC cluster with a public cloud service endpoint 

When you create a multizone VPC cluster with the public cloud service endpoint enabled, a public Ingress controller is created by default. The Ingress controller assigns publicly accessible routes for your apps and listens for requests to your apps on the public host network interface.

The following diagram shows how a Ingress controller directs network traffic from the internet to an app in a multizone, VPC cluster.

Zoom

![Expose an app in a multizone VPC cluster by using an Ingress controller](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/roks_router_vpc.png)

Figure 1. Expose an app in a multizone VPC cluster by using an Ingress controller



1. A request to your app uses the route hostname that you set up for your app.
2. A DNS service resolves the route subdomain to the VPC load balancer hostname that is assigned to the services for the Ingress controller. In VPC clusters, your Ingress controller services' external IP addresses are floating, and are kept behind a VPC-assigned hostname.
3. The VPC load balancer resolves the VPC hostname to an available external IP address of a Ingress controller service that was reported as healthy.
From the [Virtual private cloud dashboard](https://cloud.ibm.com/vpc-ext/network/vpcs), click the name of the Default Security Group for the VPC that your cluster is in.
2. Click the Rules tab.



* To create new inbound rules to control inbound traffic to your worker nodes, in the Inbound rules section, click Create.
* To create new rules to control outbound traffic to your worker nodes, in the Outbound rules section, delete the default rule that allows all outbound traffic. Then, in the Outbound rules section, click Create.





Keep in mind that in addition to any rules you create, you must also create the required [inbound and outbound rules](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-security-groupvpc-sg-inbound-outbound).





 Creating rules in the command line 

Use the IBM Cloud CLI to add inbound and outbound rules to the default security group for your cluster.



1. Install the infrastructure-service plug-in. The prefix for running commands is ibmcloud is.

ibmcloud plugin install infrastructure-service
2. Target the region that your VPC is in.

ibmcloud target -r <region>
3. Get your cluster's ID.

ibmcloud oc cluster get -c <cluster_name>
4. List your security groups and note the ID of the default security group for your VPC. Note that the default security group uses a randomly generated name and not the format kube-<cluster_ID>.

ibmcloud is sgs

Example output with the default security group for the VPC of a randomly generated name, chamomile-dislodge-showier-unfilled.

ID Name Rules Network interfaces VPC Resource group
1a111a1a-a111-11a1-a111-111111111111 chamomile-dislodge-showier-unfilled 5 2 events-vpc default
If you are not connected to the IBM Cloud private network, this option is not required because the cluster service URL is used. Before you can use the link endpoint type, make sure you set up the required Source[in your Location](https://cloud.ibm.com/docs/satellite?topic=satellite-link-endpoint-secure).
* vpe: If it is a VPC cluster, set to vpe to use the Virtual Private Endpoint gateway for your cluster context. Note you must be connected to the same VPC where the cluster is deployed through [VPC VPN connection](https://cloud.ibm.com/docs/vpc?topic=vpc-vpn-overview).



--network
: Optional: Download the Calico configuration file, TLS certificates, and permission files that are required to run calicoctl commands in your cluster. This option can't be used with the --yaml option.

--skip-rbac
: Skip adding user Kubernetes RBAC roles based on the IBM Cloud IAM service access roles to the cluster configuration. Include this option only if you [manage your own Kubernetes RBAC roles](https://cloud.ibm.com/docs/openshift?topic=openshift-usersrbac). If you use [IBM Cloud IAM service access roles](https://cloud.ibm.com/docs/openshift?topic=openshift-iam-service-access-roles) to manage all your RBAC users, don't include this option.

-q
: Optional: Do not show the message of the day or update reminders.

--yaml
: Optional: Prints the command output in YAML format.



 Example cluster config command 

ibmcloud oc cluster config --cluster my_cluster







 ibmcloud oc cluster create classic 

Classic infrastructure

Create a cluster with worker nodes on classic infrastructure.

Red Hat OpenShift on IBM Cloud clusters are created with a public only or both a public and private service endpoint.
To set up a secure connection between your cluster and an on-premises network, [configure and deploy the strongSwan IPSec VPN service](https://cloud.ibm.com/docs/containers?topic=containers-vpnvpn-setup) directly in a pod in your cluster.
* Worker nodes connected to a private VLAN only: Set up an IPSec VPN endpoint on a gateway appliance, such as a Virtual Router Appliance (Vyatta). Then, [configure the strongSwan IPSec VPN service](https://cloud.ibm.com/docs/containers?topic=containers-vpnvpn-setup) in your cluster to use the VPN endpoint on your gateway. If you don't want to use strongSwan, you can [set up VPN connectivity directly with VRA](https://cloud.ibm.com/docs/containers?topic=containers-vpnvyatta).



If you plan to connect your cluster to on-premises networks, check out the following helpful information:



* You might have subnet conflicts with the IBM-provided default 172.30.0.0/16 range for pods and 172.21.0.0/16 range for services. You can avoid subnet conflicts when you [create a cluster from the CLI](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_cluster_create) by specifying a custom subnet CIDR for pods in the --pod-subnet option and a custom subnet CIDR for services in the --service-subnet option.
* If your VPN solution preserves the source IP addresses of requests, you can [create custom static routes](https://cloud.ibm.com/docs/containers?topic=containers-static-routes) to ensure that your worker nodes can route responses from your cluster back to your on-premises network.
Configure a [DNS service that is available on the private network](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/). 2. [Enable the private ALB and create an Ingress resource](https://cloud.ibm.com/docs/containers?topic=containers-managed-ingress-setup). 3. In your private firewall, open port 80 for HTTP or port 443 for HTTPS to the IP address for the private ALB. 4. Customize ALB routing rules with [annotations](https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotations).







 Choosing a deployment pattern for VPC clusters 

Make your app accessible from only a private network by creating private NodePort, LoadBalancer, or Ingress services.

Check out the following load balancing deployment patterns for private app networking in VPC clusters:

NodePort
: Load-balancing method: Port on a worker node that exposes the app on the worker's private IP address.
: Use case: Test private access to one app or provide access for only a short amount of time. Note: You can access an app through a NodePort only if you are connected to your private VPC network, such as through a VPN connection.
: Implementation: [Create a private NodePort service](https://cloud.ibm.com/docs/containers?topic=containers-nodeport).

VPC application load balancer
: Load-balancing method:Basic load balancing that exposes the app with a private hostname.
: Use case: Quickly expose one app to a private network with a VPC application load balancer-assigned private hostname.
: Implementation: [Create a private LoadBalancer service](https://cloud.ibm.com/docs/containers?topic=containers-vpc-lbaassetup_vpc_ks_vpc_lb) in your cluster. A multizonal VPC application load balancer is automatically created in your VPC that assigns a hostname to your LoadBalancer service for your app.
You can find the hostname that is assigned to your public Ingress controller and the hostname that is assigned to your private Ingress controller by running oc get svc -n openshift-ingress and looking for the EXTERNAL IP field.

In your VPC infrastructure dashboard, the VPC load balancer reports as healthy only the two worker nodes that run the Ingress controller replica pods, because these worker nodes are configured as the listeners for the VPC load balancer. Even though only the listener worker nodes are reported as healthy, the listeners' backend pool of worker nodes is kept up-to-date by Red Hat OpenShift on IBM Cloud so that all worker nodes in your cluster can still receive requests from the VPC load balancer.







 Route resource 

To expose an app by using Route, you must create a Kubernetes service for your app and register this service with the Ingress controller by defining an Route resource. The Ingress resource is a Red Hat OpenShift resource that defines the rules for how to route incoming requests for apps.

The Route resource also specifies the path to your app services. The paths to your app services are appended to your cluster's Ingress subdomain to form a unique app URL such as mycluster-a1b2cdef345678g9hi012j3kl4567890-0000.us-south.containers.appdomain.cloud/myapp1.

One Route resource is required for each project where you have apps that you want to expose.



* If the apps in your cluster are all in the same project, you must create one Route resource to define the routing rules for the apps that you want to expose. Note that if you want to use different domains for the apps within the same project, you can create one resource per domain.
* If the apps in your cluster are in different projects, you must create one Route resource for each project to define the app's routing rules.



For more information, see [Planning networking for single or multiple projects](https://cloud.ibm.com/docs/openshift?topic=openshift-ingress-roks4multiple_projects).
5. Optional: If you selected Create VPC, complete the following steps.



1. Select a Geography and Region. To review the available regions, see [VPC multizone regions](https://cloud.ibm.com/docs/openshift?topic=openshift-regions-and-zoneszones-vpc)
2. Enter a name for your VPC.
3. Select the Resource group where you want to create your VPC.
4. Enter any tags that you want to associate with your VPC.
5. Decide whether to Allow SSH and Allow ping in the default security group.
6. Decide whether to allow access to Classic resources.
7. Review the subnet details. By default, subnets are spread evenly across zones. If you want to modify your subnets, uncheck the Create subnet in every zone box.
8. If you modify your subnet details, specify the number of IP addresses to create. VPC subnets provide IP addresses for your worker nodes and load balancer services in the cluster, so [create a VPC subnet with enough IP addresses](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-subnetsvpc_basics_subnets). Note that you can't change the number of IPs that a VPC subnet has later.
9. If you enter a specific IP range, don't use the following reserved ranges: 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16.



6. Select a standard IBM Cloud Object Storage instance where a bucket is automatically created for your cluster's internal registry.
7. Select the cluster version.
8. Optional: For the OCP entitlement section, select an entitlement for a worker pool, if you have one.



* Usually, select Purchase additional licenses for this worker pool.
You can avoid subnet conflicts when you [create a cluster from the CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clicli_cluster-create-vpc-gen2) by specifying a custom subnet CIDR for pods in the --pod-subnet option and a custom subnet CIDR for services in the --service-subnet option.

If your VPN solution preserves the source IP addresses of requests, you can [create custom static routes](https://cloud.ibm.com/docs/openshift?topic=openshift-static-routes) to ensure that your worker nodes can route responses from your cluster back to your on-premises network.

The 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16 subnet ranges are prohibited because they are reserved for Red Hat OpenShift on IBM Cloud control plane functionality.



 IBM Cloud VPC VPN 

With the IBM Cloud VPC VPN, you connect an entire VPC to an on-premises data center. This option allows you to remain VPC-native in you VPN connection setup. To get started:



1. [Configure an on-prem VPN gateway](https://cloud.ibm.com/docs/vpc?topic=vpc-vpn-onprem-example).
2. [Create a VPN gateway in your VPC, and create the connection between the VPC VPN gateway and your local VPN gateway](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-a-vpc-using-the-ibm-cloud-consolevpn-ui). If you have a multizone cluster, you must create a VPC gateway on a subnet in each zone where you have worker nodes.







 Direct Link 

With Direct Link, you can create a direct, private connection between your remote network environments and Red Hat OpenShift on IBM Cloud without routing over the public internet.
And if you need to troubleshoot issues with the cluster deployment, the bootstrap node is needed because it contains the Terraform logs for the compute and storage cluster creation. The bootstrap node should not be deleted until the cluster is no longer required.

The default VPC instance profile for the bootstrap node has been selected based on the performance of the Ansible scripts that are triggered to deploy the compute and storage cluster resources in parallel. If you choose a smaller VPC instance profile, the deployment time might be longer.

The offering enables deployment of either scratch (or ephemeral) or persistent storage, depending on application requirements. A scratch configuration uses virtual server instances with instance storage, whereas a persistent configuration uses bare metal servers with locally attached NVMe storage. If a virtual server instance with instance storage is powered off, all data stored on the instance storage volumes is rendered inaccessible after a subsequent power up of the virtual server instance. Therefore, use of scratch storage is not recommended for long running or mission-critical workloads. In addition to higher resilience, persistent storage provides higher performance and capacity than scratch storage.

The offering supports the bring-your-own-license (BYOL) model for IBM Spectrum Scale to deploy an HPC cluster on IBM Cloud. Make sure that you have sufficient software licenses to deploy the required capacity on the IBM Cloud cluster. Contact your IBM Cloud sales or support team for evaluation licenses. Or, you can also try out the scratch storage capability of the offering on IBM Cloud without a license by selecting the evaluation storage type. For more information, see [Storage types](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-storage-types).

Spectrum Scale enables all three interfaces: UI, API, and CLI. To use the API and CLI interfaces, the Terraform-based automation code is available in this [public GitHub repository](https://github.com/IBM/ibm-spectrum-scale-ibm-cloud-schematics).

The offering enables the initial Spectrum Scale-based HPC cluster creation. Any updates that are needed post-deployment regarding Spectrum Scale configuration or setup should be performed by using Spectrum Scale tools and commands.
: If you create multiple clusters that use the same subnets in one VPC, you can't use ACLs to control traffic between the clusters because they share the same subnets. You can use [Calico network policies](https://cloud.ibm.com/docs/containers?topic=containers-network_policiesisolate_workers) to isolate your clusters on the private network.

When you use the following steps to create custom ACLs, only network traffic that is specified in the ACL rules is permitted to and from your VPC subnets. All other traffic that is not specified in the ACLs is blocked for the subnets, such as cluster integrations with third party services. If you must allow other traffic to or from your worker nodes, be sure to specify those rules where noted in the following steps.

For more information, see the [VPC documentation](https://cloud.ibm.com/docs/vpc?topic=vpc-using-acls).



 Creating ACLs from the console 

For each subnet that your cluster is attached to, use the IBM Cloud VPC console to create a custom ACL with rules that limit inbound and outbound network traffic to only communication that is necessary for the cluster to function.

Looking for a simpler security setup? Leave the default ACL for your VPC as-is, and modify the [default security group](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-security-group) instead.



1. Multizone clusters only: In the [Subnets for VPC dashboard](https://cloud.ibm.com/vpc-ext/network/subnets), note the IP Range of each subnet that your cluster is attached to.
2. In the [Access control lists for VPC dashboard](https://cloud.ibm.com/vpc-ext/network/acl), click New access control list.
3. Give your ACL a name and choose the VPC and resource group that your subnets are in.
4. In the Rules section, delete the default inbound rule and outbound rule that allow all inbound and outbound traffic.
Deny ALL Any <br><br> * <br><br><br> Any <br><br> * <br><br><br> Set to bottom 



 Master IPs can be found by looking at the server: entry in the cluster's kubeconfig file (kubectl config view) and then running the nslookup or dig command to determine which IPs that it resolves to. The Master IP can be from 1 to 3 different IPs depending on the region and the control plane cluster.

* Alternatively, to allow the inbound traffic for router health checks, you can create a single inbound rule and outbound rule to allow all incoming and outgoing traffic on port 80.
7. In the Attach subnets section, choose the name of the subnet for which you created this ACL.
8. Click Create access control list.
9. Multizone clusters: Repeat steps 2 - 8 to create an ACL for each subnet that your cluster is attached to.





 Creating ACLs with the CLI 

For each subnet that your cluster is attached to, use the IBM Cloud CLI to create a custom ACL with rules that limit inbound and outbound network traffic to only communication that is necessary for the cluster to function. Note that these steps outline the minimum ACL rules that are required to allow a cluster to deploy and to have basic function; they aren't intended to cover all use cases.

Looking for a simpler security setup? Leave the default ACL for your VPC as-is, and modify the [default security group](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-security-group) instead.

Before you begin



1. Install the infrastructure-service plug-in. The prefix for running commands is ibmcloud is.

ibmcloud plugin install infrastructure-service
2. Target the region that your VPC is in.

ibmcloud target -r <region>



To create an ACL for each subnet that your cluster is attached to,



1. List your VPC subnets.