vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Team based privacy using IAM, VPC, Transit Gateway and DNS 

This tutorial may incur costs. Use the [Cost Estimator](https://cloud.ibm.com/estimator/review) to generate a cost estimate based on your projected usage.

Microservices are popular because they allow an enterprise to organize their development teams around the services they deliver. This tutorial walks you through the steps of creating infrastructure for a IBM Cloud® Virtual Private Cloud (VPC) based microservice architecture. In this architecture, VPCs are connected to each other using the IBM Cloud® Transit Gateway. A set of shared microservices is accessed through host names registered in the IBM Cloud® DNS Services. Each VPC is managed by a separate team isolated by IBM Cloud® Identity and Access Management. Optionally a IBM Cloud® Load Balancer can be used to scale out the shared microservice.



 Objectives 



* Learn how to isolate infrastructure using IAM and Resource groups
* Create the VPCs and associated resources such as subnets, network ACLs, security groups, instances.
* Address microservices by DNS name resolution using DNS Services.
* Connect VPCs via Transit Gateway.
* Transparently configure a Load Balancer for an application.





 Abstract Architecture: 

Zoom

![Architecture](https://cloud.ibm.com/docs-content/v1/content/09179fd478349810f8adb559294b68138ac393ca/hardware-firewall-shared/includes/solution-tutorials/includes/solution-tutorials/images/solution59-vpc-tg-dns-iam/simple.png)

Figure 1. Architecture diagram of the tutorial

In the diagram above the end user is accessing the applications. The applications are leveraging shared microservices. The company has separate DevOps teams that own application1, application2 and shared. A networking team focuses on connectivity and network security. The DevOps teams manage Virtual Service Instances, VSIs, used to implement the services they create and support.





 Concrete Architecture
Your vCenter Server must be configured for MS AD authentication.

You have several options for reference to your existing domain as an identity source:



* If you have connectivity to your domain controllers in or from IBM Cloud® Virtual Private Cloud, you can reference them directly.
* You can deploy read-only replica controllers in IBM Cloud VPC.
* You can add one-way trust from the domain controllers that are deployed in IBM Cloud VPC to your domain controllers.



In IBM Cloud VPC, you can deploy one or two Active Directory domain controllers by using VPC Virtual Servers Instances or by using VMs in your VMware cluster.

If you choose the option to deploy the MS AD servers on VMware, you are responsible to provide Microsoft licensing and activation.





 vSphere Single Sign On (SSO) Domain 

The vSphere Single Sign On (SSO) domain is used as the initial authentication mechanism for a single instance or multiple linked instances. The SSO domain also serves to connect a VMware instance or multiple linked instances to the MS AD server.

The following considerations must be done for SSO configuration:



* By default, vCenter Server uses the vsphere.local domain as the identity source. However, you can change it during installation.
* Select your own SSO domain to be used. For example, ibmcloud.local.
* Make the previous selections before you deploy your hosts and vCenter Server.









 Domain Name Services 

In this design, the Domain Name Services (DNS) use IBM Cloud DNS Services. The IBM Cloud DNS Services provides private DNS to VPC users and it can be used for workloads, management, and infrastructure components.

Private DNS zones are resolvable only on IBM Cloud, and only from explicitly allowed networks in an account. For shared cloud service endpoints, use the following DNS server addresses 161.26.0.10 and 161.26.0.11.

You can also use private DNS custom resolvers, which extend IBM Cloud DNS Services capability by enabling resolution of the IBM Cloud VPC hostnames from on-premises DNS resolvers. They also enable the resolution of on-premises hostnames from the IBM Cloud.
For clusters that run version 4.13 or later: If you enabled only the private cloud service endpoint during cluster creation, the virtual private endpoint of your VPC is used by default to access Red Hat OpenShift components such as the Red Hat OpenShift web console or OperatorHub. You must be connected to the private VPC network, such as through a VPN connection, to access these components or run kubectl commands on your cluster. Note that to access the console through the VPE, you must be able to access cloud.ibm.com, which requires public connectivity.



1. Set up your IBM Cloud VPC VPN and connect to your VPC through VPN.



1. Configure a [client-to-site](https://cloud.ibm.com/docs/vpc?topic=vpc-vpn-client-to-site-overview) or [site-to-site](https://cloud.ibm.com/docs/vpc?topic=vpc-vpn-onprem-example) VPN to your VPC. For example, you might choose to set up a client-to-site connection with an OpenVPN Client.
2. For client-to-site VPN setups, you must specify the IBM Cloud VPC Private DNS service addresses when you provision the VPN server as mentioned in the [considerations](https://cloud.ibm.com/docs/vpc?topic=vpc-client-to-site-vpn-planningexisting-vpc-configuration-considerations). You must also create a VPN route after the VPN server is provisioned, with the destination 161.26.0.0/16 and action translate.
3. For site-to-site VPN setups, you must follow the [Accessing service endpoints through VPN guide](https://cloud.ibm.com/docs/vpc?topic=vpc-build-se-connectivity-using-vpn) and configure the IBM Cloud VPC Private DNS service addresses.
4. Verify that you are connected to the VPC through your IBM Cloud VPC VPN connection.



2.
Creating single-tier VPC for SAP on IBM Cloud® VPC with Terraform 

Terraform on IBM Cloud® enables predictable and consistent provisioning of IBM Cloud VPC infrastructure resources so that you can rapidly build complex, cloud environments. IBM Cloud VPC infrastructure consists of SAP certified hardware using Intel Xeon CPUs and additional Intel technologies.

For more information about Terraform on IBM Cloud®, see [Terraform on IBM Cloud getting started tutorial](https://cloud.ibm.com/docs/ibm-cloud-provider-for-terraform?topic=ibm-cloud-provider-for-terraform-getting-started).

To create resources with Terraform, you use Terraform configuration files that describe the IBM Cloud resources that you need and how you want to configure them. Based on your configuration, Terraform creates an execution plan and describes the actions that need to be run to create the resources. You can review the execution plan, change it, or run the plan. When you change your configuration, Terraform on IBM Cloud can determine what changed and create incremental execution plans that you can apply to your existing IBM Cloud resources.



 Script files 

The configuration and script files are provided on the GitHub repository [https://github.com/IBM-Cloud/sap-infra-anydb-single/tree/main/cli](https://github.com/IBM-Cloud/sap-infra-anydb-single/tree/main/cli).

For single-tier virtual private cloud on SAP, you modify the:



* terraform.tfvars file to add your IBM Cloud API-key
* input.auto.tfvars file to customize the resources for your solution. You specify zones, resource names, and SSH keys.



All of the other configuration files are provided and do not need to be modified.

The IBM Cloud Provider plug-in for Terraform on IBM Cloud uses these configuration files to provision a VPC in your IBM Cloud account.





 What is created 

A VPC is a private space in IBM Cloud where you can run an isolated environment with custom network policies. The variables that you define are used by the scripts to provision the following virtual private cloud infrastructure resources for you:
Existing free tier clusters will be allowed to finish their 30-day trial window. If you want to try IBM Cloud Kubernetes Service, [contact IBM Sales](https://www.ibm.com/account/reg/us-en/signup?formid=MAIL-wcp).

You can have 1 free cluster at a time in IBM Cloud Kubernetes Service, and each free cluster expires in 30 days. Free clusters are ideal for testing out Kubernetes deployments and getting familiar with the IBM Cloud Kubernetes Service API, CLI, and console tools. After you are done testing your free cluster, you can [copy your configuration files and redeploy them to a standard cluster](https://cloud.ibm.com/docs/containers?topic=containers-update_appcopy_apps_cluster).





 What container platforms are available for my cluster? 

With IBM Cloud, you can create clusters for your containerized workloads from two different container management platforms: the IBM version of community Kubernetes and Red Hat OpenShift on IBM Cloud. The container platform that you select is installed on your cluster master and worker nodes. Later, you can [update the version](https://cloud.ibm.com/docs/containers?topic=containers-updateupdate) but can't roll back to a previous version or switch to a different container platform. If you want to use multiple container platforms, create a separate cluster for each.

For more information, see [Comparison between Red Hat OpenShift and community Kubernetes clusters](https://cloud.ibm.com/docs/containers?topic=containers-cs_ovopenshift_kubernetes).

Kubernetes
: [Kubernetes](https://kubernetes.io/) is a production-grade, open source container orchestration platform that you can use to automate, scale, and manage your containerized apps that run on an Ubuntu operating system.
You can create virtual server instances for IBM Cloud® Virtual Private Cloud in Dallas (us-south), Washington DC (us-east), London (eu-gb), Sydney (au-syd), Tokyo (jp-tok), Osaka (jp-osa), Frankfurt (eu-de), Madrid (eu-es), Toronto (ca-tor), and São Paulo (br-sao).
* Can I use existing virtual server instances from my classic infrastructure with an IBM Cloud VPC?

You can migrate a virtual server instance from the classic infrastructure to a VPC. You need to create an image template, export it to IBM Cloud Object Storage, and then customize the image to meet the requirements of the VPC infrastructure. For more information, see [Migrating a virtual server from the classic infrastructure](https://cloud.ibm.com/docs/vpc?topic=vpc-migrate-vsi-to-vpc).
* What virtual server families are supported in IBM Cloud VPC?

Currently, public virtual servers in the balanced, memory, and compute families are supported. For more information, see [Profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-profilesprofiles).
* What do I do if an instance is in a bad state, such as continually starting or stopping?

You can issue a command to force the instance to stop. Use the IBM Cloud CLI to obtain the instance ID, and then run the following command, ibmcloud is instance-stop --no-wait -f. When the instance is stopped, you can either restart it or delete it.
* When I attempt to update my Ubuntu image with apt, I receive an error about the grub menu.lst file. How do I fix it?

Edit the file "/boot/grub/menu.lst" by changing groot=LABEL... into groot=(hd0). Then, run following command, sudo update-grub-legacy-ec2.
Zoom

![Architecture](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution40-vpc-public-app-private-backend/Architecture.svg)

Figure 1. Architecture diagram of the tutorial



1. The admin (DevOps) sets up the required infrastructure (VPC, subnets, security groups with rules, VSIs) on the cloud.
2. The internet user makes an HTTP/HTTPS request to the web server on the frontend.
3. The frontend requests private resources from the secured backend and serves results to the user.







 Before you begin 



* Check for user permissions. Be sure that your user account has sufficient permissions to create and manage VPC resources. See the list of [required permissions](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-user-permissions-for-vpc-resources) for VPC.
* You need an SSH key to connect to the virtual servers. If you don't have an SSH key, see [the instructions](https://cloud.ibm.com/docs/vpc?topic=vpc-ssh-keys) for creating a key for VPC.







 Step 1: Create a Virtual Private Cloud and subnets 

To tighten the security of your servers, it is recommended to only allow connections to the ports required by the applications deployed on the servers. In this tutorial, the application will be a web server, thus it will only need to allow inbound connections on port 80.

To perform maintenance tasks on these servers such as installing software, performing operating system upgrades, you will go through a bastion host. A bastion host is an instance that is provisioned with a public IP address and can be accessed via SSH. Once set up, the bastion host acts as a jump server allowing a secure connection to instances provisioned in the VPC.

In this section, you will create the VPC and the bastion host.
[Architecture](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution62-vpc-scaling-dedicated/architecture_diagram.svg)

Figure 1. Architecture diagram of the tutorial



1. The frontend app deployed on VSI(s) communicates to the backend app via the private load balancer.
2. The backend app securely communicates with the cloud services via a virtual private endpoint (VPE).
3. As the load on the application increases, scaling for VPC is enabled and dynamically adds or removes VSIs based on metrics like CPU, RAM, etc., or through scheduled scaling.
4. As the scope expands, dedicated host isolates and performs heavy computation on the data. Resize the instance on the dedicated host by updating the profile based on your requirement. Also, expand the block storage volume capacity.
5. All instances communicate with IBM Cloud services over the private backbone using a virtual private endpoint (VPE). See the [About virtual private endpoint gateways](https://cloud.ibm.com/docs/vpc?topic=vpc-about-vpe) topic for more details.







 Before you begin 

The tutorial requires:



* An IBM Cloud [billable account](https://cloud.ibm.com/docs/account?topic=account-accounts).







 Step 1: Provision required cloud services 

In this section, you will create the following cloud services required for the application using IBM Cloud Schematics: IBM Cloud Databases for PostgreSQL and IBM Cloud Object Storage.



1. Navigate to [Schematics Workspaces](https://cloud.ibm.com/schematics/workspaces), click on Create workspace.



1. Under the Specify Template section, provide https://github.com/IBM-Cloud/vpc-scaling-dedicated-host under GitHub or GitLab repository URL.
2.
Step 1: Create a VPC 

To create your own IBM Cloud VPC in region 1, complete these steps:



1. Go to the VPC overview page and click Get Started.
2. Select Create.
3. Under New virtual private cloud section:



* Enter vpc-region1 as the name for your VPC.
* Select a Resource group.
* Optionally, add Tags to organize your resources.



4. The default access control list (ACL) (Allow all) is appropriate for your VPC.
5. Clear Allow SSH and Allow ping from the Default security group and leave classic access cleared. You add SSH access to the maintenance security group later. The maintenance security group must be added to an instance to allow SSH access from the bastion server. Ping access is not required for this tutorial.
6. Leave Create a default prefix for each zone checked.
7. Under New subnet for VPC:



* Enter vpc1-region1-zone1-mgmt as your subnet's unique name.
* Select a Resource group.
* Select a location and zone 1 for example: Dallas and Dallas 1.
* Select the wanted number of IP addresses.



8. Leave the access control list set to Use VPC default.
9. Leave the public gateway set to Detached.
10. Click Create virtual private cloud.







 Step 2: Create subnets in the same availability zone of region 1 



 Create subnets in availability zone 1 

You create three extra subnets for your first availability zone (VPC-region1-zone1) and use the VPC you created in Step 1:



* vpc-region1-zone1-ui
* vpc-region1-zone1-application
* vpc-region1-zone1-db



To create the subnets:



1. Click Subnet
2. Click Create.
3. Enter vpc-region1-zone1-ui as a unique name for your subnet.
4. Select vpc-region1 as the VPC.
5.
For more information, see [Getting started with Virtual Private Cloud (VPC)](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-started).



1. Create a VPC.
2. Create subnets.
3. Order virtual server instance.



* SSH key
* Operating system name (Linux or Windows and their respective versions)
* Security groups
* Secondary volume (optional)



Encrypted volumes are not supported.







 Option 2: Auto-provision 



 Setting up a cloud user 



1. Log in to the RackWare web console.
2. In the RackWare web console, navigate to Configuration > Clouduser.
3. When you add a cloud user, enter a name and select IBM Cloud VPC for the Cloud Provider. Select the region where you want to auto-provision the virtual server instance, and enter your IBM Cloud API key.
4. Click Add.







 Creating a wave and replication 

A wave contains a single host or multiple hosts that will be migrated. For this migration, you need to create one or more waves, provide information about the hosts in the wave, and then start the wave.



1. In the RackWare web console, nagivate to Replication > Waves.
2. When you create a wave, select Target Type as Autoprovision.
3. Enter source and target details.



If source machine has Linux operating system and it has a boot volume greater than 100 GB then leave ‘Provision disk’ textbox empty and select Convert to LVM option. This will create additional disk of required size and will convert all eligible volumes to LVM on the target. In case of Windows, no action is required. It will be taken care automatically by RMM.



4. After you enter your source and target information, you need to provide your IBM Cloud VPC information.
5. From the edit option in Actions menu of your source, select the IBM Cloud VPC Options tab, enter the relevent information, and click Modify.
6. Run the replication.
* CLI
* API
* Terraform







 Accessing virtual private endpoints from an on-premises network using Direct Link or from another VPC using Transit Gateway 

For Virtual Private Endpoint enabled services, you can access IBM Cloud resources over a private network from your on-premises location through IBM Cloud Direct Link, or from another VPC using IBM Cloud Transit Gateway. By using an endpoint gateway, traffic between your VPC and IBM Cloud services does not leave the private network.

IBM Cloud Virtual Private Endpoint (VPE) for VPC enables you to access supported IBM Cloud services remotely by using the IP addresses of your choice, which are allocated from a subnet within your VPC.

You should put VPEs in a separate, dedicated subnet that is not associated with your workloads. If you want to access a VPE from another subnet, ensure that you configure your network ACLs or security groups to allow for cross-subnet access. For example, make sure that you open the proper ports within your ACLs or security groups to allow access.

Only TCP services are supported.



 Architecture 

The following topology illustrates remote, direct access to a VPE in a VPC through Direct Link and Transit Gateway. The VPEs are created in a central place and custom resolvers are used to resolve the VPE.

The topology demonstrates that you can access a VPE from a remote client that is connected through Direct Link or through Transit Gateway. You must provision a custom resolver in the VPC where you created VPE gateways for the Domain Name Server (DNS) resolution. When accessing VPE from a remote client through Direct Link, you must configure DNS forwarding rules for the private endpoints that you want to access through VPE, to the custom resolver provisioned in the VPC where VPE gateways are created. When accessing VPE from a remote client through Transit Gateway, you must also provision a custom resolver in the VPC where the client is running, and configure DNS forwarding rules to the custom resolver provisioned in the VPC where VPE gateways are created.

Zoom

!
Complete the following steps to set up better control over granting access to compute resources.

IBM Cloud Kubernetes Service supports only trusted profiles for versions 1.21 and newer. Free IBM Cloud Kubernetes Service clusters create only earlier versions. You have the choice to create clusters with an earlier version with the standard plan, so be sure to select version 1.21 or newer.



1. Create a trusted profile by specifying your account ID.

curl -X POST 'https://iam.cloud.ibm.com/v1/profiles' -H 'Authorization: Bearer TOKEN' -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{
"name": "My Nice Profile",
"description": "My Nice Profile - desc",
"account_id": "ACCOUNT_ID"
}'

In the description, provide a list of actions available for this trusted profile.
2. Create a claim rule for your trusted profile that follows the principle of least privilege by specifying conditions that target only the resources that need access. Conditions apply to all existing future resources. You can also create a direct link with specific, existing resources.

Create conditions
: For compute resources, set the type attribute to Profile-CR. For more information, see the [IAM Identity Services API](https://cloud.ibm.com/apidocs/iam-identity-token-apicreate-claim-rule-request).

curl -X POST 'https://iam.cloud.ibm.com/v1/profiles/PROFILE_ID/rules' -H 'Authorization: Bearer TOKEN' -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{
"type": "Profile-CR",
"conditions": [
{
"claim": "namespace",
"operator": "EQUALS",
"value": ""default123""
}
]
}'

There is a limit of 20 claim rules per trusted profile.

Create a direct link
Bring Your Own IP Address 

This tutorial describes the use of Classic Infrastructure. Most workloads can be implemented using [IBM Cloud® Virtual Private Cloud](https://cloud.ibm.com/docs/vpc) resources. Use IBM Cloud VPC to create your own private cloud-like computing environment on shared public cloud infrastructure. A VPC gives an enterprise the ability to define and control a virtual network that is logically isolated from all other public cloud tenants, creating a private, secure place on the public cloud. Specifically, you can [bring your own subnet](https://cloud.ibm.com/docs/vpc?topic=vpc-configuring-address-prefixes) IP address range to a IBM Cloud VPC.

This tutorial may incur costs. Use the [Cost Estimator](https://cloud.ibm.com/estimator/review) to generate a cost estimate based on your projected usage.

This tutorial presents a brief overview of BYOIP implementation patterns that can be used with IBM Cloud and a decision tree for identifying the appropriate pattern when realizing the secure enclosure as described in the [Isolate workloads with a secure private network](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-secure-network-enclosure) tutorial. Setup may require additional input from your onsite network team, IBM Cloud technical support or IBM Services.

Bring Your own IP (BYOIP) is a frequent requirement where its desired to connect existing client networks to infrastructure provisioned on the IBM Cloud. The intent is typically to minimize change to the clients network routing configuration and operations with the adoption of a single IP address space based on the clients existing IP addressing scheme.



 Objectives 



* Understand BYOIP implementation patterns
* Select implementation pattern for IBM Cloud.







 IBM Cloud IP addressing 

The IBM Cloud utilizes a number of private address ranges, most specifically 10.0.0.0/8, and in some cases these ranges may conflict with existing client networks. Where address conflicts exist, there are a number of patterns that support BYOIP to allow interoperability with the IBM Cloud network.



* Network Address Translation