vania-janet/multiturn-rag-retrieval-data huggingface dataset:
pytest -m 'vpe or vpedns'



It can take a few tries for the DNS names to be resolved accurately. So try the test at least three times. All tests should pass except the enterprise to spoke VPE tests:





 Step 5: Enterprise to Spoke Virtual Private Endpoint Gateways 

The current configuration does not support enterprise -> spoke VPE traffic. The VPE return traffic does not use spoke egress route table.

It is possible to improve the results by adding the enterprise address prefixes to the transit VPC. This will allow the spoke VPE return traffic to learn the enterprise routes and forward traffic through the through the Transit Gateway to the transit VPC. However, it will be delivered to the transit VPC zone of the matching address prefix. Enterprise -> spoke cross zone will not work.

The enterprise DNS resolution must resolve the fully qualified name to the IP address of the VPE to the zone it has been assigned in the transit VPC. Configuring this is beyond the scope of this tutorial.



1. Edit config_tf/terraform.tfvars.



* Change the value enterprise_phantom_address_prefixes_in_transit = true.



2. Configuration changes require all current layers to be re-applied

./apply.sh : :
3. Verify that all VPEs can be accessed from all test instances.

Your expected results are: enterprise -> spoke vpe cross zone will be FAILED:

pytest -m 'vpe or vpedns'







 Production Notes and Conclusions 

The [VPC reference architecture for IBM Cloud for Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-about) has much more detail on securing workloads in the IBM Cloud.

Some obvious changes to make:



* CIDR blocks were chosen for clarity and ease of explanation. The Availability Zones in the Multi zone Region could be 10.1.0.0/10, 10.64.0.0/10, 10.128.0.0/10 to conserve address space.
The default value is 600 seconds.
* Transport protocol - Select one of the following protocols and enter a valid VPN port (1 - 65535).



* UDP is a connectionless protocol that performs error checking but discards erroneous packets. Faster than TCP.
* TCP is a connection-oriented protocol with error checking and error recovery. More reliable than UDP.



* Tunnel mode - Specify one of the following:



* Full tunnel - All traffic flows through the VPN interface to the VPN tunnel.
* Split tunnel - Private traffic flows through the VPN interface to the VPN tunnel, and public traffic flows through the existing LAN interface.

Zoom

![VPN additional configuration section](https://cloud.ibm.com/docs-content/v1/content/f02b9c9adcbb7328d95a45e105cec371d50f15eb/vpc/images/vpn-additional-configuration.png)

VPN additional configuration section











 Creating a VPN server from the CLI 

Before you begin, make sure to [set up your CLI environment](https://cloud.ibm.com/docs/vpc?topic=vpc-infrastructure-cli-plugin-vpc-reference).

To create a VPN server by using the CLI, enter the following command:

ibmcloud is vpn-server-create --subnets SUBNETS --client-ip-pool CLIENT_IP_POOL --cert CERT (--client-auth-methods certificate | username | certificate,username | username,certificate) [--client-ca CLIENT_CA] [--client-crl CLIENT_CRL] [--client-dns CLIENT_DNS] [--client-idle-timeout CLIENT_IDLE_TIMEOUT] [--enable-split-tunnel false | true] [--port PORT] [--protocol udp | tcp] [--security-group SECURITY_GROUP1 --security-group SECURITY_GROUP2 ...]
Architecture pattern for multisite NSX-T topologies 

VMware vCenter Server® instances offer a standard NSX-T topology with a single NSX-T edge cluster, which includes a single Tier-0 (T0) and Tier-1 (T1) Gateways. You have several options to build and customize multisite overlay topologies. You can manually provision new NSX-T edge clusters and deploy new Tier-0 (T0) and Tier-1 (T1) Gateways after initial deployment by following VMware NSX-T™ documentation.

This section provides a few examples on how these topologies can be customized for your needs.

The automated deployment provides you a single NSX-T workload edge cluster with two edge transport nodes. Multisite topologies require additional NSX-T edge transport nodes and clusters. You can manually deploy these after deployment by using IBM Cloud private IP addresses for edge management and TEPs.



 Multisite – single tenant 

Multisite – single tenant is also a common use case and network deployment pattern. This topology provides seamless site recovery by using dynamic routing protocols, such as BGP. The vCenter Server automation does not deploy multisite topology automatically. However, you can customize the default single-site topology after initial vCenter Server deployment. When you have the compute capacity in the required data center, you can manually deploy the required additional edge nodes. Then, create a new edge cluster for the Tier-0 and Tier-1 Gateways through NSX-T. This overlay topology is highly scalable and it is possible to automate, for example, by using Ansible and Terraform.

The following diagram shows an example of a multisite – single tenant network topology. It consists of a two layer Tier-0 design with three edge clusters. Two edge clusters in the two data centers or zones in a multizone region, and one edge cluster deployed across the multizone region with edge nodes in each participating zone or data center.

Zoom

!
l7PoolUuid = 'set me'

 New Layer 7 pool values to be updated
l7Pool = {'loadBalancingAlgorithm': 'LEASTCONNECTION'}

 New Layer 7 Health monitor values to be updated
l7HealthMonitor = {'urlPath': '/index'}

 New Layer 7 session affinity values to be updated.
 If not given it deletes the existing session affinity
 If given and session affinity doesn't exist, it creates one.
l7SessionAffinity = {'type': 'SOURCE_IP'}

client = SoftLayer.Client()
networkLBaaSL7PoolService = client['SoftLayer_Network_LBaaS_L7Pool']

try:
result = networkLBaaSL7PoolService.updateL7Pool(l7PoolUuid, l7Pool,
l7HealthMonitor, l7SessionAffinity)
pprint(result)
except SoftLayer.SoftLayerAPIError as e:
print("Unable to update a Layer 7 pool along with health monitoring and session affinity: %s, %s"
% (e.faultCode, e.faultString))
Show more





 Add servers to a Layer 7 Pool 

import SoftLayer
from pprint import pprint

 UUID of the L7 pool to which members should be added.
l7PoolUuid = 'set me'

 Back-end servers to be added
memberInstances = [
{
'address': '10.131.11.46',
'port': 80,
'weight': 10
},
{
'address': '10.130.188.130',
'port': 81,
'weight': 11
}
]

client = SoftLayer.Client()
networkLBaaSL7MemberService = client['SoftLayer_Network_LBaaS_L7Member']

try:
NSX-T T0 Logical Router needs its own VLAN interfaces for its uplinks.







 Naming conventions 

The following naming conventions are used for deployment. You might use your own naming convention if preferred.



Table 6. NSX-T design naming convention

 Description Naming Standard 

 Management VMs vpc-nsxt-0 <br>vpc-nsxt-1 <br>vpc-nsxt-2 
 Uplink profiles vpc-esxi-vpc-profile <br>vpc-edge-vpc-profile 
 NIOC profiles vpc-cluster-nioc-vpc-profile 
 Gateway cluster profiles vpc-zone-cluster-edge-cluster-profile 
 Transport zones nsx-overlay-transportzone <br>nsx-vlan-transportzone 
 Segments vpc-zone-customer-t0-172-16-16-0 <br>vpc-zone-customer-t1-192-168-0-0 <br>vpc-zone-t0-private-vlanid <br>vpc-zone-t0-public-vlanid <br>vpc-zone-edge-tep 
 Tier-0 and Tier-1 gateways vpc-zone-T0 <br>vpc-zone-T1 





 Transport zones 

A Transport Zone is a container that defines the potential reach of Transport Nodes, Hosts, or Edges. Default NSX-T transport zones are used in this architecture.



Table 7. NSX-T transport zones

 Transport node type Naming Standard 

 Overlay nsx-overlay-transportzone 
 VLAN nsx-vlan-transportzone 







 Transport nodes 

Transport nodes define the physical server objects or VMs that participate in the virtual network fabric. Uplink profiles provide a shared configuration for each transport node that uses it.



Table 8. NSX-T transport nodes

 Transport node type Uplink profile IP assignment 

 ESXi vpc-esxi-vpc-profile Static IP list 
 Gateway cluster Overlay : vpc-edge-vpc-profile <br>VLANs : vpc-edge-vpc-profile Static IP list
You are responsible for deploying the edge transport nodes in colocation on your own x86 hardware and for creating the NSX-T edge cluster and configuring Tier-0 (T0) Gateway on the edge cluster.

You can use Gateway Appliance or vCenter Server gateway cluster with Juniper vSRX or other device as part of the solution. This is optional.



 Deploying Direct Link with NSX-T edge cluster in colocation 

The following diagram presents an overview of an architecture pattern for deploying Direct Link with NSX-T edge cluster in colocation.

Zoom

![Direct Link with NSX-T edge cluster in colocation](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/vmwaresolutions/images/arch-pattern-vcs-nsx-t-direct-link-edge.svg)

Figure 1. Direct Link with NSX-T edge cluster in colocation

This architecture pattern deployment is summarized as follows:



1. vCenter Server instance is deployed at IBM Cloud classic infrastructure. Two IBM Cloud private VLANs and one IBM Cloud public VLAN (optional) are deployed. Each of these VLANs host multiple subnets. You can see the details through IBM Cloud for VMware Solutions portal.
2. NSX-T T0 is deployed with two interfaces - private and public (optional). If you opt for a public one, this interface is attached to your Public VLAN and has direct internet access. Your T0's private interface is attached to the Private VLAN and it uses IBM Cloud portable private IP.
3. If vCenter Server gateway cluster with vSRX (or other third-party device) or IBM Cloud Gateway Appliance is deployed to your classic infrastructure, you must configure your vCenter Server instance Private Primary VLAN. It is routed through the vSRX or Gateway Appliance and allows TEP traffic through it.
4. Create a Direct Link at your IBM Cloud data center or zone location and attach your classic network as a connection.
Zoom

![Setting up NSX-T L2 bridge in NSX-V environment](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/vmwaresolutions/images/v2t-nsx-t-l2-bridge-1.svg)

Figure 1. Setting up NSX-T L2 bridge in NSX-V environment



1. Register the vCenter appliance of the vCenter server with NSX-V environment as a compute manager in the NSX-T Manager. For more information, see [Add a Compute Manager](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/installation/GUID-D225CAFC-04D4-44A7-9A09-7C365AAFCA0E.html).
2. On the vCenter Server with NSX-V instance, complete the following steps:



* Order a private portable subnet on the Private VLAN for use by the NSX-T Edge node VMs.
* Create a distributed port group on the private vDS for use by the management and the overlay transport zone.



3. Using NSX-T Manager, the NSX-T Edge VMs are deployed onto hosts in the vCenter Server with NSX-V environment and configured:



* Configure an IP Pool that contains IP addresses from the private portable subnet that were ordered previously. For example, TEP-NSX-V-IP-POOL.
* The Edge Node VM requires two host switches:



* nsxHostSwitch overlay - is associated with an overlay transport zone that is shared with the NSX-T ESXi transport nodes and uses the IP pool that was previously configured.
* nsxHostSwitch vlan - is associated with a VLAN transport zone and is used to enable bridging the virtual wire that is connected to this interface to an NSX-T segment.
This change results in the IP address of the firewall-router changing from the firewall-router instance used earlier to the IP address of the NLB. The IP address change need to be applied to a number of VPC route table routes in the transit and spoke VPCs. It is best to apply all of the layers previously applied:
2. Apply all the layers through the all_firewall_asym_tf layer:

./apply.sh : all_firewall_asym_tf



Observe the changes that were made:



1. Open the [Load balancers for VPC](https://cloud.ibm.com/vpc-ext/network/loadBalancers).
2. Select the load balancer in zone 1 (Dallas 1/us-south-1) it has the suffix fw-z1-s3.
3. Note the Private IPs.



Compare the Private IPs with those in the transit VPC ingress route table:



1. Open the [Virtual Private Clouds](https://cloud.ibm.com/vpc-ext/network/vpcs).
2. Select the transit VPC.
3. Click on Manage routing tables.
4. Click on the tgw-ingress routing table. Notice the Next hop IP address matches one of the NLB Private IPs



Verify resiliency:



1. Run the spoke 0 zone 1 tests:

pytest -k r-spoke0-z1 -m curl
2. Open the [Virtual server instances for VPC](https://cloud.ibm.com/vpc-ext/compute/vs)
3. Stop traffic to the 0 firewall instance by specifying a security group that will not allow inbound port 80. Locate the instance with the suffix fw-z1-s3-0 and open the details view:



1. Scroll down and hit the pencil edit next to the Network Interface
2. Uncheck the x-fw-inall-outall
3.
firewall_nlb = true
number_of_firewalls_per_zone = 2

This change results in the IP address of the firewall-router changing from the firewall-router instance used earlier to the IP address of the NLB. The IP address change need to be applied to a number of VPC route table routes in the transit and spoke VPCs. It is best to apply all of the layers previously applied:
2. Apply all the layers through the all_firewall_asym_tf layer:

./apply.sh : all_firewall_asym_tf



Observe the changes that were made:



1. Open the [Load balancers for VPC](https://cloud.ibm.com/vpc-ext/network/loadBalancers).
2. Select the load balancer in zone 1 (Dallas 1/us-south-1) it has the suffix fw-z1-s3.
3. Note the Private IPs.



Compare the Private IPs with those in the transit VPC ingress route table:



1. Open the [Virtual Private Clouds](https://cloud.ibm.com/vpc-ext/network/vpcs).
2. Select the transit VPC.
3. Click on Manage routing tables.
4. Click on the tgw-ingress routing table. Notice the Next hop IP address matches one of the NLB Private IPs



Verify resiliency:



1. Run the spoke 0 zone 1 tests:

pytest -k r-spoke0-z1 -m curl
2. Open the [Virtual server instances for VPC](https://cloud.ibm.com/vpc-ext/compute/vs)
3. Stop traffic to the 0 firewall instance by specifying a security group that will not allow inbound port 80. Locate the instance with the suffix fw-z1-s3-0 and open the details view:



1. Scroll down and hit the pencil edit next to the Network Interface
2.
When the initial NSX Manager is deployed into the host and cluster, you must register the vCenter as the compute manager to facilitate the deployment of other NSX Managers. Public Gateway attached to the management subnet can be used to download updates for the NSX Managers.

For more information on deploying NSX Managers, see [VMware NSX-T Data Center Documentation](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/index.html).





 Host transport nodes 

For NSX-T, each ESXi host must be set as a Transport Node so that it becomes capable of participating in an NSX-T Data Center overlay or NSX-T Data Center VLAN networking. As of version 3, NSX-T can run on the vSphere Distributed Switch version 7.0, and each host if provisioned as described in the [Physical infrastructure design](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vpc-ryo-infrastructure-physical) can be added as a Transport Node.

The following table lists the required VLAN interfaces for NSX-T VMKs for each host. These interfaces are always local to the host and are not allowed to float. They are provisioned on the vpc-tep-subnet. You can provision these VLAN interfaces in IBM Cloud VPC either when you provision the host, or when you configure and deploy NSX-T.



Table 3. Host management networks and VMkernel adapters

 Interface name Interface type VLAN ID Subnet Allow float VMkernel adapter Distributed Port Group name 

 vlan-nic-tep-vmk10 vlan 400 vpc-tep-subnet false vmk10 dpg-tep 



Host TEP VLAN ID is defined in the host transport profile. The dpg-tep creation is optional, and for consistency only. It might help you to identify the VLAN ID used when operating the environment later.





 Edge transport nodes and gateway cluster
vCenter Server IP address 
 vCenter Server user 
 vCenter Server password 
 NSX Transport Zone transport-1 



 Allow self-signed certificates on PowerCLI
Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore

 Connect to NSX Manager
Connect-NsxServer -vCenterServer <IP_Address> -User <UserName> -Password '<Password>'

 Create logical switches
Get-NsxTransportZone transport-1 | new-nsxlogicalswitch -Name OpenShift-LS -Description "OpenShift network"
Get-NsxTransportZone transport-1 | new-nsxlogicalswitch -Name OpenShift-Transit -Description "OpenShift transit network"
Get-NsxTransportZone transport-1 | new-nsxlogicalswitch -Name OpenShift-HA -Description "OpenShift DLR HA network"

 Disconnect
Disconnect-NsxServer
Show more





 Related links 



* [Red Hat OpenShift Bastion node setup](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-openshift-runbook-runbook-bastion-intro)
* [Red Hat OpenShift 4.7 user provider infrastructure installation](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-openshift-runbook-runbook-install-intro)
volumeMounts:
- mountPath: /host/var/log
name: host-log
volumes:
- name: host-log
hostPath:
path: /var/log
Show more
5. Create the DaemonSet in your cluster when the Transport endpoint is not connected. error occurs.

oc create -f ./inspectnode-ds.yaml
6. Get the name of the inspectnode pod.

oc get pods -n default -l app=ibm-inspectnode -o wide
7. Copy the logs from inspectnode pod to your local machine. In the copy command, give your directory a name so that you can identify which node the logs are from. For example, node-one.log. You might also use the node IP in the directory name, for example: 10.XXX.XX.XXX.log.

oc cp <inspectnode_pod_name>:/host/var/log ./<node_name>.log
8. Review the syslog and the s3fslog for information about the Transport endpoint error. [Open a support ticket](https://cloud.ibm.com/docs/openshift?topic=openshift-get-helphelp-support) and share the log files that you gathered.
9. Delete the ibm-inspectnode DaemonSet that you deployed.

oc delete daemonset ibm-inspectnode
Apply the all_firewall_asym layer:

./apply.sh all_firewall_asym_tf
2. Run the test suite.

Your expected results are: all tests PASSED, run them in parallel:

pytest -n 10 -m curl



All traffic between VPCs is now routed through the firewall-routers.







 Step 2: High Performance High Availability (HA) Firewall-Router 

To prevent a firewall-router from becoming the performance bottleneck or a single point of failure it is possible to add a VPC Network Load Balancer to distribute traffic to the zonal firewall-routers to create a Highly Available, HA, firewall-router. Check your firewall-router documentation to verify it supports this architecture.

Zoom

![High Availability Firewall](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/vpc-transit/vpc-transit-ha-firewall.svg)

High Availability Firewall

This diagram shows a single zone with a Network Load Balancer (NLB) configured in route mode fronting two firewall-routers. To see this constructed it is required to change the configuration and apply again.



1. Change these two variables in config_tf/terraform.tfvars:

firewall_nlb = true
number_of_firewalls_per_zone = 2

This change results in the IP address of the firewall-router changing from the firewall-router instance used earlier to the IP address of the NLB. The IP address change need to be applied to a number of VPC route table routes in the transit and spoke VPCs. It is best to apply all of the layers previously applied:
2. Apply all the layers through the all_firewall_asym_tf layer:

./apply.sh : all_firewall_asym_tf



Observe the changes that were made:



1.
GRE tunnel connectivity issues 

If BGP is not established over the GRE tunnel, first try pinging the BGP peer IP address to ensure layer 2 connectivity exists over the GRE tunnel.

If the BGP settings are correct, then check that the MTU setting on your device matches the MTU for the GRE gateway.

Otherwise, if BGP can utilize OpenSent or OpenConfirm, check the BGP multi-hop settings on your device. Some GRE tunnel configurations may not adjust the BGP TCP TTL. As a workaround, set your multi-hop to a value of 16, which will force the TTL value of the BGP packet to expire after 16 routed hops.
When authentication is completed, the system passes the code to the openVPN client for authentication.



Zoom

![VPN client authentication section](https://cloud.ibm.com/docs-content/v1/content/f02b9c9adcbb7328d95a45e105cec371d50f15eb/vpc/images/vpn-client-authentication.png)

VPN client authentication section



9. In the Security groups section, select at least one security group. To configure one or more security groups and their rules, see [Configuring ACLs and security groups for use with VPN](https://cloud.ibm.com/docs/vpc?topic=vpc-acls-security-groups-vpn).

Zoom

![VPN security groups section](https://cloud.ibm.com/docs-content/v1/content/f02b9c9adcbb7328d95a45e105cec371d50f15eb/vpc/images/vpn-security-groups.png)

VPN security groups section

You can attach security groups after provisioning from the VPN server's details page (Security Groups tab).
10. In the Additional configuration section, specify the following information:



* DNS server IP address - Optionally, specify one or two DNS server IP addresses for domain name resolution.
* Idle timeout session (seconds) - If the VPN server has no traffic before this interval lapses, it is automatically disconnected. This value can range between 0 to 28800 seconds. The default value is 600 seconds.
* Transport protocol - Select one of the following protocols and enter a valid VPN port (1 - 65535).



* UDP is a connectionless protocol that performs error checking but discards erroneous packets. Faster than TCP.
* TCP is a connection-oriented protocol with error checking and error recovery. More reliable than UDP.



* Tunnel mode - Specify one of the following:
They are attached to a portable subnet in a private primary VLAN on the IBM Cloud private network. If you provision your instance with public interfaces, the access to public network is provided through public uplinks. They are attached to a portable subnet on a public VLAN on the IBM Cloud public network. The uplinks have IP addresses specific to this data center and they cannot move between the data centers.
4. You must manually deploy two additional edge cluster transport nodes. Also, you must create a new edge cluster for your Tier-0 and Tier-1 Gateways in the second IBM Cloud data center hosts.
5. In the second IBM Cloud data center, the access to private network is provided through private uplinks. They are attached to a portable subnet on a private primary VLAN on the IBM Cloud private network. If you provision your instance with public interfaces, the access to public network is provided through public uplinks. They are attached to a portable subnet on a public VLAN on the IBM Cloud public network. The uplinks have IP addresses specific to this data center and they cannot move between the data centers.
6. You must manually deploy two or more edge cluster transport nodes in the regional edge cluster, at least one in each zone or data center for high availability. Then, create a new edge cluster by using these edge transport nodes, and create your regional Tier-0 in this cluster. You can select which data center (or edge node) is your preferred Tier-0 path in the regional cluster.
7. Create an overlay transit and uplinks in Tier-0 Gateways into the transit segment. Establish dynamic routing between the Tier-0 Gateways and BGP. This is the preferred routing protocol.
8. Then, create your Tier-1 Gateways in the regional edge cluster. You can select which data center (or edge node) is your preferred Tier-1 path in the regional cluster.
9. You can attach your segments into this level of Tier-1 Gateways. Also, you can create multiple segments and advertise them through Tier-1 Gateway to north bound Tier-0 Gateways.