vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* [IBM® Event Streams for IBM Cloud® to feed event data to apps](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-aboutabout)





 Context 

Shipping company increases availability of worldwide systems for business partner ecosystem



* Regional differences for shipping logistics made it difficult to keep up with growing number of partners in multiple countries. An example is the unique regulations and transit logistics, where the company must maintain consistent records across borders.
* Just-in-time data meant that the worldwide systems must be highly available to reduce lags in transit operations. Time tables for shipping terminals are highly controlled and sometimes inflexible. Web usage is growing, so instability might cause a poor user experience.
* Developers needed to constantly evolve apps, but traditional tools slowed their ability to deploy updates and features frequently.







 Solution 

The shipping company needs to cohesively manage shipping time tables, inventories, and customs paperwork. Then, they can accurately share the location of shipments, shipping contents, and delivery schedules to their customers. They’re taking the guess work out of when a good (such as an appliance, clothing, or produce) arrives so that their shipping customers can communicate that information to their own customers.

The solution is made up of the following primary components.



1. Streaming data from IoT devices for each shipping container: manifests and location
2. Customs paperwork that’s digitally shared with applicable ports and transit partners, including access control
3. App for shipping customers that aggregates and communicates arrival information for shipped goods, including APIs for shipping customers to reuse shipment data in their own retail and business-to-business apps



For the shipping company to work with global partners, the routing and scheduling systems required local modifications to fit each region’s language, regulations, and unique port logistics. IBM Cloud Kubernetes Service offers global coverage in multiple regions, including North America, Europe, Asia, and Australia so that the apps reflected the needs of its partners, in each country.

IoT devices stream data that IBM® Event Streams for IBM Cloud® distributes to the regional port apps and associated Customs and Container manifest data stores. IBM® Event Streams for IBM Cloud® is landing point for IoT events.
The same network subnet considerations for transit gateway connections also apply to Direct Link connections. To ensure successful connectivity, do not use prefixes in your Direct Link connected network that overlap with other connections.





 VPC connection consideration 

IBM Cloud VPC permits the use of RFC-1918 and IANA-registered IPv4 address space, privately within your VPC, with some exceptions in the IANA special-purpose ranges, and select ranges assigned to IBM Cloud services. When using IANA-registered ranges within your enterprise, and within VPCs in conjunction with IBM Cloud Transit Gateway, custom routes must be installed in each zone. For more information, see [Routing considerations for IANA-registered IP assignments](https://cloud.ibm.com/docs/vpc?topic=vpc-interconnectivityrouting-considerations-iana).





 VPC interconnection consideration 

You can create a single transit gateway or multiple transit gateways to interconnect more than one IBM Cloud VPCs. You can also connect your IBM Cloud classic infrastructure to a transit gateway to provide seamless communication with classic infrastructure resources. For more information, refer to [Interconnecting VPCs](https://cloud.ibm.com/docs/vpc?topic=vpc-interconnectivity&interface=cliinterconnecting-vpcs).





 Routing considerations 



* All connections to a transit gateway are connected to each other, so carefully consider all resources that you want to interconnect before deciding whether local or global routing is right for each gateway.

Traffic from either routing option does not leave the private IBM Cloud network and is optimized for performance.
* If you plan to use your gateway to connect VPCs in the same multi-zone region ([MZR](https://cloud.ibm.com/docs/overview?topic=overview-locationsmzr-table)), use local routing to provide connectivity to all accessible resources within the same MZR; for example, us-south (Dallas).

Zoom

!
How do I verify and deal with connectivity issues on my transit gateways? 

There are several issues that can cause problems when attempting to communicate between resources connected to your IBM Cloud® Transit Gateway.

  What’s happening 

You may receive error messages that state you need to check whether the resource you are requesting to access exists, and to review your network topology.

  Why it’s happening 

Known issues include incorrect permissions, overlapping VPC prefixes and classic infrastructure subnets, networking interface card routing, access control lists, firewalls and gateway appliances, and routing in classic VSIs.

  How to fix it 

To fix the connectivity issues you're encountering, review the following information topics:



*  [Connectivity issue basics](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-connectivity-issue-basics)
*  [Working with firewalls and gateway appliances](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-firewalls-gateways)
*  [Dealing with overlapping VPC prefixes and classic infrastructure subnets](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-overlapping-vpc-prefixes-and-classic-subnets)
*  [Working with Network Interface Card Routing](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-tg-nic)
*  [Routing in classic virtual server instances](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-routing-classic-vsi)
*  [GRE tunnel connectivity issues](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-gre-tunnel-conn-issues)
Similarly the CIDR 192.168.0.0/24 covers the enterprise and is passed through the Transit Gateway to the spokes as an advertised route.

Egress routes in the spokes route traffic to the firewall-router. Ingress routes in the transit route enterprise <-> spoke traffic through the firewall-router.





 Step 1: Provision initial VPC resources routing all intra VPC traffic through the firewall-router 

Often an enterprise uses a transit VPC to monitor the traffic with the firewall-router. In part one only enterprise <-> spoke traffic was flowing through the transit firewall-router. This section is about routing all VPC to VPC traffic through firewall-router.

This diagram shows the traffic flow implemented in this step:

Zoom

![Traffic flow](https://cloud.ibm.com/docs-content/v1/content/6a28e4ae2660b609a18e90e1239ed772f80373d6/transit-gateway/includes/solution-tutorials/includes/solution-tutorials/images/vpc-transit/vpc-transit-part2-fw.svg)

Traffic flow

All traffic between VPCs will flow through the firewall-router:



* enterprise <-> spoke.
* enterprise <-> transit.
* transit <-> spoke.
* spoke <-> spoke in different VPC.



Traffic within a VPC will not flow through the firewall.

If continuing from part one make special note of the configuration in the terraform.tfvars: all_firewall = true.



 Apply Layers 



1. The companion [GitHub Repository](https://github.com/IBM-Cloud/vpc-transit) has the source files to implement the architecture. In a desktop shell clone the repository:

git clone https://github.com/IBM-Cloud/vpc-transit
cd vpc-transit
2. The config_tf directory contains configuration variables that you are required to configure.
* The subnet CIDRs further divide the /24 into /26.



The subnets in the transit and spoke are for the different resources:



* worker - network accessible compute resources VPC instances, load balancers, [Red Hat OpenShift](https://www.ibm.com/cloud/openshift), etc. VPC instances are demonstrated in this tutorial.
* dns - DNS Services location appliances used in part two.
* vpe - [VPE for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-about-vpe) used in part two.
* firewall - firewall-router VPC instances.







 Step 1: Provision VPC network resources 



1. The companion [GitHub Repository](https://github.com/IBM-Cloud/vpc-transit) has the source files to implement the architecture. In a desktop shell clone the repository:

git clone https://github.com/IBM-Cloud/vpc-transit
cd vpc-transit
2. The config_tf directory contains configuration variables that you are required to configure.

cp config_tf/template.terraform.tfvars config_tf/terraform.tfvars
3. Edit config_tf/terraform.tfvars and use the comments in that file as your guide.
4. Since it is important that each layer is installed in the correct order and some steps in this tutorial will install multiple layers a shell command ./apply.sh is provided. The following will display help:

./apply.sh
5. You could apply all of the layers configured by executing ./apply.sh : :. The colons are shorthand for first (or config_tf) and last (vpe_dns_forwarding_rules_tf). The -p prints the layers:

./apply.sh -p : :

It will look something like:
=============== 12 failed, 4 passed, 276 deselected in 72.80s (0:01:12) ================
Show more



A change to the network configuration can take a couple of test runs for the underlying VPC network system to become consistent. If you do not see the expected results initially be prepared to run the test again a couple of times.

The r- and l- stand for right and left. The middle part of the name identifies enterprise, transit, spoke0, spoke1, ... The z1, z2, ... identify the zone. The test will SSH to the left instance. On the left instance the connectivity to the right instance is attempted. The test_curl performs a curl connectivity on the left instance to the right instance.

The test test_curl[l-enterprise-z1 -> r-transit-z1]:



1. SSH to a test instance in enterprise zone 1.
2. Execute a curl to transit zone 1.
3. Assert the return string contains the ID of transit zone 1 to mark pass or fail.



The README.md in the companion [GitHub Repository](https://github.com/IBM-Cloud/vpc-transit) has more details and the source code.





 Step 4: Connect Transit and Spokes via Transit Gateway 

Provision a IBM Cloud Transit Gateway to connect the transit <-> spoke(s) and spoke(s) <-> spoke(s).

Zoom

![Spoke gateway](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/vpc-transit/vpc-transit-spoke-tgw.svg)

Spoke gateway



1. Apply the layer:

./apply.sh transit_spoke_tgw_tf
2.
* [Direct link (2.0)](https://cloud.ibm.com/docs/dl?topic=dl-get-started-with-ibm-cloud-dl)
* [VPNs](https://cloud.ibm.com/docs/vpc?topic=vpc-vpn-overview)
* [Transit Gateway](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-ordering-transit-gateway)





To improve data transfer rate, adjust bandwidth allocation of RMM server. To know how to change bandwidth allocation, see [Adjusting bandwidth allocation using the UI](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-virtual-server-instances&interface=uiadjusting-bandwidth-allocation-ui).





 Migration considerations 



 Capabilities 



* Application and environmental settings are preserved
* Data sync (data drift) before final cut-over is supported
* Data migration (secondary and block volume)



VPC does not have support for snapshot, replication, and shared volume. You can manage these solutions through the native OS capabilities, tools, or third party of your choice.





 Limitations 



* New IP address (original IP is not preserved).
* Classic add-ons are not carried over and will be lost.
* File-based storage migration is not supported.
* Hardware configuration is not brought over (RAID configuration).
* Bare metal with GPU is not supported due to no GPU support on VPC.
* Not an OS replatforming (Windows to Linux or vice versa, changing linux distribution, or major software upgrade). Software upgrades should be handled through the normal process.









 Validating your migration 

After your server migration, you will want to validate your compute resources, such as applications and data, and update or validate the following:



* Validate or update your FQDN/Hostname.
* Update your yum repos, application settings, and configuration file.
* Update licensing where applicable.
* Remove RMM SSH key from target after final validation.
* For enterprise appdomain.com will forward to the transit.
* For transit the fully qualified DNS name of the Databases for Redis instance will be forwarded to the spoke instance that owns the Databases for Redis instance.
* For spoke_from -> spoke_to access to Redis the spoke_from needs the DNS name for the Databases for Redis instance. The fully qualified Redis name in spoke_from DNS instance will be forwarded to the transit.
* The transit forward fully qualified Redis names to the corresponding spoke.



Zoom

![Enabling DNS for virtual private endpoints](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/vpc-transit/vpc-transit-dns-vpe.svg)

Enabling DNS for virtual private endpoints

The diagram uses transit-.databases.appdomain.cloud to identify the database in the transit instead of the fully qualified name like 5c60b3e4-1920-48a3-8e7b-98d5edc6c38a.c7e0lq3d0hm8lbg600bg.private.databases.appdomain.cloud.
3. Apply the vpe_dns_forwarding_rules_tf layer:

./apply.sh vpe_dns_forwarding_rules_tf
4. Verify that all VPEs can be accessed from all test instances.

Your expected results are: enterprise -> spoke vpe tests will be FAILED:

pytest -m 'vpe or vpedns'



It can take a few tries for the DNS names to be resolved accurately. So try the test at least three times. All tests should pass except the enterprise to spoke VPE tests:





 Step 5: Enterprise to Spoke Virtual Private Endpoint Gateways 

The current configuration does not support enterprise -> spoke VPE traffic. The VPE return traffic does not use spoke egress route table.
Starting on the left notice that the enterprise DNS forwarding rule will look to the transit for the DNS zones: x-transit.com, x-spoke0.com, and x-spoke1.com. The transit DNS instance can resolve x-transit.com and has its own forwarding rules to the enterprise and spokes to resolve the rest. Similarly the spokes rely on the transit DNS instance to resolve the enterprise, transit and the other spokes.
8. Optionally explore the other DNS instances and find similarly named DNS zones and A records for the other test instances.







 DNS Testing 

There is a set of curl DNS tests that are available in the pytest script. These tests will curl using the DNS name of the remote. There are quite a few so run the tests in parallel:

pytest -n 10 -m dns







 Step 4: Virtual Private Endpoint Gateways 

VPC allows private access to IBM Cloud Services through [Virtual Private Endpoint (VPE) for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-about-vpe). The VPEs allow fine grain network access control via standard IBM Cloud VPC controls:



* [Security Groups](https://cloud.ibm.com/docs/vpc?topic=vpc-using-security-groups).
* [VPC Network Access Control Lists](https://cloud.ibm.com/docs/vpc?topic=vpc-using-acls).
* [Routing tables and routes](https://cloud.ibm.com/docs/vpc?topic=vpc-about-custom-routes).



Zoom

![Adding virtual private endpoint gateways](https://cloud.ibm.com/docs-content/v1/content/6a28e4ae2660b609a18e90e1239ed772f80373d6/transit-gateway/includes/solution-tutorials/includes/solution-tutorials/images/vpc-transit/vpc-transit-vpe.svg)

Adding virtual private endpoint gateways



1.
It's the time that is elapsed between the first bytes received and the moment the service receives the last byte of the request body. <br>transfer- The data transmission time to transfer the full response from the service to the client. <br>response- The total active time for the HTTP request, between the moment the service received the first byte of the request header and the emission of the last byte of the response body. 



The request and transfer timings are included in response.

IBM Cloud Log Analysis offers [searching](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-view_logsview_logs_step6) and [filtering](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-view_logsview_logs_step5) to help you navigate your logs. [Export](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-exportexport) and [archive](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-archivingarchiving) are available so you can customize retention (and cost) for your use case.
* Classic access VPCs cannot be attached to a transit gateway. How can I connect and access classic resources in those VPCs?

Although [classic-access VPCs](https://cloud.ibm.com/docs/vpc?topic=vpc-setting-up-access-to-classic-infrastructure) cannot be attached to a transit gateway, access to classic resources and classic-access VPC resources can be achieved by adding the classic infrastructure connection to a transit gateway. For more information, see [Classic infrastructure connection considerations](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-helpful-tipsclassic-infrastructure-connection-considerations).
* How can "VPC peering" be achieved on IBM Cloud?

IBM Cloud Transit Gateway can be used to connect multiple VPCs to each other. As such, connecting/peering two VPCs is just a part of the functionality that the transit gateway service offers. IBM Cloud does not provide a standalone VPC peering service or capability.
* Can I connect a VPN or a direct link to a transit gateway?

IBM Cloud Direct Link can be connected to either a local or global transit gateway.

Currently, you cannot connect a VPN to a transit gateway.
* Can I create a global transit network using the IBM Cloud Transit Gateway?

IBM Cloud Transit Gateway enables standard IP routing between networks (for example, global VPCs) that are connected to it. You can add additional functionality by configuring IBM or third-party virtual network functions, such as VPN, NAT, and firewalls, within one or more of the interconnected networks (for instance, using the "Transit VPC" concept).
* How can I guarantee one of my clients is not going to impact the others?

Capacity management handles the overall available capacity on the transit gateway and is subject to our weekly capacity management review. When the device reaches roughly a 50% load, we augment the connectivity to the device.
* What scalability options do I have for my transit gateway? Does it manage itself? How do I know if it's reaching maximum capacity?

The IBM Cloud infrastructure manages all transit gateways. There are no scalability options available.
Planning considerations when using AS prepends with VPC connections 

Currently, VPC networking doesn't consider the AS path length when selecting the best route for network traffic. However, you can use a transit gateway between the direct link and VPC in certain topologies to achieve the same outcome. Also, if prefix values are repeated across different AS prepends (of the same policy), the first instance of the prefix value sets the prefix length; the rest are ignored.

Keep in mind that AS prepends currently have no effect on the routing between VPCs and direct links.



 Examples 

The following deployment topologies illustrate various AS Prepend scenarios when connecting to VPCs.



1. VPC chooses a route regardless of the AS path length.

Zoom

![AS prepends have no effect on the routing between VPCs and direct links](https://cloud.ibm.com/docs-content/v1/content/b698ea8660e1f51664f7479f870bbf8840335d06/dl//images/asprepends_2.png)

Figure 1. AS prepends have no effect on the routing between VPCs and direct links.
2. VPC routes to the transit gateway and the transit gateway considers the AS path length when choosing the route to on-premises.

Zoom

![AS prepends are considered during route determination](https://cloud.ibm.com/docs-content/v1/content/b698ea8660e1f51664f7479f870bbf8840335d06/dl//images/asprepends_1.png)

Figure 2. AS prepends are considered during route determination.
3. The local transit gateway is preferred over the global transit gateway.

Zoom

!
* [IBM® Event Streams for IBM Cloud® to feed event data to apps](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-aboutabout)





 Context 

Shipping company increases availability of worldwide systems for business partner ecosystem



* Regional differences for shipping logistics made it difficult to keep up with growing number of partners in multiple countries. An example is the unique regulations and transit logistics, where the company must maintain consistent records across borders.
* Just-in-time data meant that the worldwide systems must be highly available to reduce lags in transit operations. Time tables for shipping terminals are highly controlled and sometimes inflexible. Web usage is growing, so instability might cause a poor user experience.
* Developers needed to constantly evolve apps, but traditional tools slowed their ability to deploy updates and features frequently.







 Solution 

The shipping company needs to cohesively manage shipping time tables, inventories, and customs paperwork. Then, they can accurately share the location of shipments, shipping contents, and delivery schedules to their customers. They’re taking the guess work out of when a good (such as an appliance, clothing, or produce) arrives so that their shipping customers can communicate that information to their own customers.

The solution is made up of the following primary components.



1. Streaming data from IoT devices for each shipping container: manifests and location
2. Customs paperwork that’s digitally shared with applicable ports and transit partners, including access control
3. App for shipping customers that aggregates and communicates arrival information for shipped goods, including APIs for shipping customers to reuse shipment data in their own retail and business-to-business apps



For the shipping company to work with global partners, the routing and scheduling systems required local modifications to fit each region’s language, regulations, and unique port logistics. Red Hat OpenShift on IBM Cloud offers global coverage in multiple regions, including North America, Europe, Asia, and Australia so that the apps reflected the needs of its partners, in each country.

IoT devices stream data that IBM® Event Streams for IBM Cloud® distributes to the regional port apps and associated Customs and Container manifest data stores. IBM® Event Streams for IBM Cloud® is landing point for IoT events.
Core <br><br> * [VPC infrastructure services](https://cloud.ibm.com/docs/vpc?topic=vpc-manage-security-compliance)[1]<br><br><br> <br><br> * [Satellite](https://cloud.ibm.com/docs/satellite?topic=satellite-complianceplatform-compliance)<br><br><br> 
 Containers <br><br> * [Red Hat OpenShift on IBM Cloud](https://cloud.ibm.com/docs/openshift?topic=openshift-manage-security-compliance)<br> * [Container Registry](https://cloud.ibm.com/docs/openshift?topic=openshift-manage-security-compliance)<br><br><br> <br><br> * [Red Hat OpenShift on IBM Cloud](https://cloud.ibm.com/docs/openshift?topic=openshift-manage-security-compliance)[2]<br> * [Container Registry](https://cloud.ibm.com/docs/openshift?topic=openshift-manage-security-compliance)<br><br><br> 
 Networking <br><br> * [VPC infrastructure services](https://cloud.ibm.com/docs/vpc?topic=vpc-manage-security-compliance)<br> * [Direct Link](https://cloud.ibm.com/docs/dl?topic=dl-manage-security-compliance)<br> * [Transit Gateway](https://cloud.ibm.com/docs/transit-gateway?topic=transit-gateway-manage-security-compliance)<br><br><br>