vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Shared - create VSI and block devices in the shared VPC. Create DNS records for shared services.
* Application1 - create VSI and block devices in the application1 VPC
* Application2 - create VSI and block devices in the application2 VPC.





 IAM Conceptual 



 Network Team 

A conceptual team ownership model was implemented. The network team administers all the network resources and hence most of what is shown in the diagram.

Zoom

![Architecture focusing on the Network team](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution59-vpc-tg-dns-iam/network.png)

Architecture focusing on the Network team





 Shared Team 

The shared team creates the VSI in its isolated VPC. In addition, the team needs to write records into the DNS service since the IP addresses of the VSIs are determined at creation time. Operator access to the VPC, subnets and security groups are required to create a VSI.

Zoom

![Architecture focusing on the Shared team](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution59-vpc-tg-dns-iam/shared.png)

Architecture focusing on the Shared team

The application teams needs the same access as the shared team except manager access to the DNS Services.

Application team access:

Zoom

![Architecture focusing on the Application team](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution59-vpc-tg-dns-iam/app1.png)

Architecture focusing on the Application team
If your app is packaged in a Docker image, you can use Docker commands to check the memory usage of your app.



1. Create a container locally that runs your Docker image.

docker run IMAGE_NAME
2. Get a list of the containers to get a container ID.

docker ps
3. Check the statistics of the running container.

docker stats CONTAINER_ID
4. Review the memory usage value for the container. If the value does not fit within the system limits, adjust your script.
5. After you are done reviewing the information, you can stop the running container.

docker stop CONTAINER_ID
6. Remove the container.

docker rm CONTAINER_ID





 Special considerations for memory usage with Node.js runtime actions 

 What’s happening 

In the case where an action consumes more memory than requested, the action is terminated and the following log information is displayed:

2019-10-22T10:00:50.509Z stderr: Killed
2019-10-22T10:00:50.510Z stderr: The action did not initialize or run as expected. Log data might be missing.

 Why it’s happening 

When Cloud Functions runs successive invocations of the same action, the optimizations that are performed by Cloud Functions might consume more memory than expected to improve run times.

For example, when Cloud Functions runs actions, Linux containers are used for the processes. To speed up the process, new containers are not created each time that your action runs ("cold"), but instead, existing containers that ran your action before ("warm") are reused. So when your action completes, the container "freezes" to pause all processes and then "wakes" when your action is rerun.

This approach affects garbage collection. When you run a Node.js action, garbage is created on the heap. This garbage is accumulating over each warm reuse of the action process. However, because the Node.js process pauses between invocations, garbage collection does not run reliably.

 How to fix it
CM-7 - Least Functionality 



 Control requirements 

The organization:

CM-7 (a)
: Configures the information system to provide only essential capabilities; and

CM-7 (b)
: Prohibits or restricts the use of the following functions, ports, protocols, and/or services: [IBM Assignment: limiting, disabling, and/or controlling services, features, applications, functions, ports, and protocols not explicitly required to support business functionality].





 IBM Cloud for Financial Services profile 

The rules related to this control that follow are part of the IBM Cloud for Financial Services v1.2.0 profile in [IBM Cloud® Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-getting-started).



Rules for CM-7 in IBM Cloud for Financial Services v1.2.0 profile

 Requirement ID Rules
Subnets provide network connectivity to your cluster components by assigning IP addresses to them.

The following subnets are automatically provisioned on the default public and private VLANs.



 Public VLAN subnets 



* The primary public subnet determines the public IP addresses that are assigned to worker nodes during cluster creation. Multiple clusters on the same VLAN can share one primary public subnet.
* The portable public subnet is bound to one cluster only and provides the cluster with 8 public IP addresses. 3 IPs are reserved for IBM Cloud infrastructure functions. 1 IP is used by the default public Ingress ALB and 4 IPs can be used to create public network load balancer (NLB) services or more public ALBs. Portable public IPs are permanent, fixed IP addresses that can be used to access NLBs or ALBs over the internet. If you need more than 4 IPs for NLBs or ALBs, see [Adding portable IP addresses](https://cloud.ibm.com/docs/containers?topic=containers-subnetsadding_ips). Note that these IP addresses are intended for use in IBM Cloud Kubernetes Service. Do not use these IP addresses for other purposes outside of IBM Cloud Kubernetes Service.







 Private VLAN subnets 



* The primary private subnet determines the private IP addresses that are assigned to worker nodes during cluster creation. Multiple clusters on the same VLAN can share one primary private subnet.
* The portable private subnet is bound to one cluster only and provides the cluster with 8 private IP addresses. 3 IPs are reserved for IBM Cloud infrastructure functions. 1 IP is used by the default private Ingress ALB and 4 IPs can be used to create private network load balancer (NLB) services or more private ALBs. Portable private IPs are permanent, fixed IP addresses that can be used to access NLBs or ALBs over a private network. If you need more than 4 IPs for private NLBs or ALBs, see [Adding portable IP addresses](https://cloud.ibm.com/docs/containers?topic=containers-subnetsadding_ips). Note that these IP addresses are intended for use in IBM Cloud Kubernetes Service.
CKM_EC_KEY_PAIR_GEN [Standards for Efficient Cryptography (SEC) curves](https://www.secg.org/sec2-v2.pdf) <br><br> * secp256k1<br><br><br> 
 CKM_EC_KEY_PAIR_GEN [Edwards curves](https://tools.ietf.org/html/rfc8032) <br><br> * Ed25519<br><br><br> 









 Performing cryptographic operations with GREP11 functions 

You can perform cryptographic operations by calling GREP11 functions that are defined based on the EP11 implementation of the PKCS #11 specification. The following function descriptions are created based on the

[PKCS #11 specification](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html), with notes specific to EP11. All parameter definitions are in the original form of EP11. For more information about EP11, see

[Enterprise PKCS #11 (EP11) Library structure](https://www.ibm.com/security/cryptocards/pciecc4/library).

EP11 function parameters are mapped to the protocol buffer types that can be found in the following functions. You can learn more about protocol buffer types in [Google Developers](https://developers.google.com/protocol-buffers/docs/proto3scalar).

Because the EP11 library is a subset of the PKCS #11 API library, and GREP11 functions are variants from the corresponding EP11 functions, the corresponding functions of EP11 and PKCS #11 are also listed in the GREP11 function tables for your reference.

GREP11 supports any programming language with a gRPC library. At the current stage, only code snippets or examples for Golang and JavaScript are included in the API reference. The content is enriched in later phases.
Virtual appliances are created for custom resolvers to serve DNS queries, or for [global load balancer health checks](https://cloud.ibm.com/docs/dns-svcs?topic=dns-svcs-global-load-balancersadd-a-health-check) to send probes to your origin servers that monitor their health status. The virtual appliance is fully managed by DNS Services, however, you can go to the Subnets for VPC dashboard and view the details of a subnet to see the reserved IPs of that subnet. You can see which IP address is bound to the network interface on the virtual appliance managed by DNS Services.

Zoom

![Reserved IP](https://cloud.ibm.com/docs-content/v1/content/d56dbd8ab7d14086029681ee71679e14aea53723/dns-svcs/images/virtual_appliance_reserved_ip.png)

Figure 1. Reserved IP for virtual appliance managed by DNS Services

From the Targeted resource column, you can view which reserved IP is bound to a DNS service instance that has a virtual appliance connected to your subnet. It is recommended that you keep Auto-release enabled (the default setting). With Auto-release, this IP address automatically releases to the IP address pool after the virtual appliance is deleted, as part of the deletion process for custom resolvers and global load balancer pools.

Disabling Auto-release can disrupt DNS Services recovery operations.





 Custom resolver status 

The status of a newly-created custom resolver is initially Critical because the resolver location is not yet enabled. The status changes to Healthy after the resolver location changes to Up.

The following status definitions apply to the resolver locations:



* Up - when the resolver location is functioning.
* Down - when the resolver location is not functioning.



The following status definitions apply to the custom resolver:



* Healthy - when all resolver locations are Up, the status is Healthy.
It is recommended to instead [use context-based restrictions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-setting-a-firewallsetting-cbr) for controlling access based on network information.





 Cloud Functions and Code Engine 

Configuring replication does not provide a [trigger for Cloud Functions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-triggers) or Code Engine events at this time, but object writes and deletes will create Object:Write and Object:Delete notifications for both the source and target buckets. These events are annotated with a notifications.replication_type field that indicates if the event triggered a sync, or was triggered by a sync.







 Replicating existing objects 

A replication rule can only act on objects that are written after the rule is configured and applied to a bucket. If there are existing objects in a bucket that should be replicated, the replication processes needs to be made aware of the existence of the objects. This can be easily accomplished by using the PUT copy operation to copy objects onto themselves.

This process will reset some object metadata, including creation timestamps. This will impact lifecycle policies and any other services that use creation or modification timestamps (such as content delivery networks). Ensure that any disruptions that may arise from resetting object metadata are dealt with appropriately.

The process involves:



1. Creating a list of all the objects in a bucket that should be subject to replication rules,
2. Iterating over that list, performing a PUT copy operation on each object with the source being identical to the target of the request.



This example will only replicate the new version of the object created by the PUT copy request. In order to replicate all versions of the object, it would be necessary to copy each individual version as well.

The following example is written in Python, but the algorithm could be applied in any programming language or context.

import os
import sys
import ibm_boto3
from ibm_botocore.config import Config

 Create client connection
vSphere components 

The IBM Cloud® for VMware® Regulated Workloads is built with vSphere technology, which includes the VMware ESXi™ hypervisors and vCenter appliance.



 Management cluster 

The management software stack consists of vCenter Server, AD/DNS, and VMware Aria® Operations™ tooling. vCenter Server manages all hosts in the vCenter Server instance. AD/DNS provide Domain Name Services (DNS) and management authentication services, a local NTP source, and role-based access control (RBAC) for access to the vCenter, VMware Aria operations tooling, and NSX-T™ administration portal. The VMware Aria operations tooling includes VMware Aria Operations Manager, and VMware Aria Operations™ for Logs. These components collectively provide a native console for vSphere operations, ability to automate management of the cloud platform, centralized log collection and analysis, and network visibility and optimization up to the hypervisor. SaaS consumer workloads are not monitored by default.

Within the Management cluster, shared storage is used to provide enhanced resiliency to the management stack. The use of shared storage provides rapid restoration of management components if an ESXi host is lost. vSAN based storage is used to keep all management stack data in the account boundary. vSAN requires a minimum cluster size of four ESXi hosts.

The vCenter appliance provides the necessary interfaces to the virtualization administrators, which include a robust RESTful API. Resilience of the vCenter appliance is delivered by using vSphere native DRS functions such as vMotion. More management components are deployed to the management cluster but only the vCenter appliance is responsible for direct operation of the platform.



 Active Directory and DNS 

Active Directory serves to authenticate accesses to manage the VMware® instance only and not to house users of the workloads in the deployed instances. The forest root domain name of the Active Directory server equals to the Domain Name Services (DNS) domain name that you specify during initial deployment.

Domain name services (DNS) in this design support cloud management and infrastructure components only. The DNS zone files are also replicated on the Active Directory servers.
Startup latency Medium Scheduled start Low 
 Termination Run-continuously Run-to-completion Run-to-completion 
 Invocation On request or permanently running Scheduled On request, instant 
 Programming Model Container based build and execution Container based build and execution Language specific source code files and dependency meta data 
 Parallelism Parallel execution, flexible Low to medium parallel execution High parallel execution 
 Scale-out Based on number of requests Based on job workload definition Based on events or direct invocations 
 Optimized for Long running highly complex workload and on-demand scale-out Scheduled/planned workloads with high resource demands Startup time and rapid scale-out 







 What are key features of working with Functions? 

Review the following topics to learn more about working with IBM Code Engine Functions.



* [Isolation](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionscefun-isolation)
* [Logging](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionsfunctions-logging)
* [Runtimes](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionsfunctions-runtime)
* [Running Functions](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionscefun-runfun)
* [Security](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionscefun-security)
* [Invocation concurrency and scaling of Function instances](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionsfunctions-concur-ce)
* [Packaging your source code for a Function](https://cloud.ibm.com/docs/codeengine?topic=codeengine-cefunctionsfunctions-packaging)





 Isolation 

Code Engine is a multi-tenant, regional service where tenants share the same network and compute infrastructure. In particular, the network and compute infrastructure are shared resources and some management components are common to all tenants.
Glossary of HCX components and terms 

VMware HCX™ consists of a cloud side (target or virtual data center environment) and one or more clients (source). An instance of HCX must be deployed per vCenter, even if the vCenters where HCX is deployed are linked in the same SSO domain on the client or cloud side. Configurations that are supported by HCX are, one-to-one, one-to-many, many-to-one, and many-to-many.



 Target side and client side 

HCX has the concept of cloud side (target or virtual data center environment) and client side (source).



* Target side - HCX Cloud is pre-deployed and configured with the network and compute profiles ready for service mesh creation.
* Client side - Any vSphere instances that meet the prerequisites for installation and operation. The client side of HCX is the primary that controls the cloud side secondary instance through its vCenter web client user interface snap-in.







 HCX Managers 



* The cloud side - HCX Cloud is configured to listen for incoming client-side registration, management, and control traffic.
* The client-side - HCX Connector is a client-side specific OVA image file that provides the user interface functions for managing and operating HCX. The client-side HCX Manager is responsible for registration with the HCX Cloud and creating a management plane between the client and cloud side. Furthermore, it is responsible for deploying a service mesh on the client side and instructing the cloud side to do the same.







 Service Mesh Components 

HCX Service Mesh components are responsible for creating the data and control planes between client and cloud side. Deployed as virtual machines (VMs) in mirrored pairs, the service mesh consists of the following components:



* Interconnect Appliance (HCX-IX) - The interconnect appliance creates encrypted tunnels that support vMotion and replication (bulk migration) traffic.
* WAN Optimizer Appliance (HCX-WAN) - HCX includes an optionally deployed Silver Peak™ WAN optimization appliance. It is deployed as a VM appliance.
Business Support Services for IBM Cloud (BSS) BSS is used to access information about the IBM Cloud account, service subscription, service usage, and billing. 
 IBM Cloud® Certificate Manager This service manages the TLS certificates that are used for the service broker. 
 IBM Cloud® Log Analysis with LogDNA The Hyper Protect Virtual Servers service broker sends service logs to IBM Cloud Log Analysis with LogDNA. These logs are monitored and analyzed by the service team to detect service issues and malicious activities. 
 IBM Cloud® Activity Tracker Hyper Protect Virtual Servers integrates with IBM Cloud Activity Tracker to forward audit events to the IBM Cloud Activity Tracker service. 
 IBM Cloud® Monitoring The Hyper Protect Virtual Servers service broker sends service metrics to IBM Cloud Monitoring. These metrics are monitored by the service team to identify capacity and performance issues of the service. 
 IBM Cloud® Identity and Access Management To authenticate requests to the service and authorize user actions, Hyper Protect Virtual Servers uses Cloud Identity and Access Management (IAM) operating system and service access roles. For more information about the IAM permissions that are required to work with the service, see [Managing access for Hyper Protect Virtual Servers](https://cloud.ibm.com/docs/hp-virtual-servers?topic=hp-virtual-servers-iam-hpvs). 
 IBM Cloud® Kubernetes Service Hyper Protect Virtual Servers is using the IBM Cloud Kubernetes Service to host the service broker and the dashboard UI. 
 IBM Cloud® Databases for MongoDB Hyper Protect Virtual Servers uses the IBM Cloud Databases for MongoDB to store the metadata about virtual servers to manage those containers. 







 Dependencies on third party services 

Review the list of third party services that Hyper Protect Virtual Servers connect to over the public network.



Table 2. Dependencies on third party services

 Service name Description 

 Akamai, Cloudflare Akamai and Cloudflare are used as the primary providers for DNS, global load balancing, and web firewall functionality in Hyper Protect Virtual Servers.
* Security group rules that control traffic between the network interfaces in port group 111 aren't applied. If you need security group rules, you need to use separate VLAN IDs for the VLAN interfaces.



3. In a distributed vSwitch topology, to enable vMotion in a specific port group, you need to make sure that the VLAN ID of this port group is included in the VLAN allowlist of all the bare metal servers.

For example, in a distributed vSwitch topology with two bare metal servers, the PCI interface of "bare-metal-server-1" has a VLAN allowlist of [111, 222, 333] and the PCI interface of "bare-metal-server-2" has a VLAN allowlist of [333, 444, 555]. Only VMs with the VLAN ID 333 can migrate between the two servers through vMotion.

The PCI interfaces don’t need to be in the same subnet.
4. A bare metal server can have only one Uplink that is set on a standard vSwitch or distributed vSwitch. Otherwise, the network might not work properly.









 Mapping network concepts between bare metal servers and VMware vSphere 

Bare Metal Servers for VPC fully supports VMware vSphere networking functions. To set up networks in the vSphere environment, you need to first understand the mapping of the networking concepts between your bare metal server and vSphere.

The following table describes the mapping of network concepts between Bare Metal Servers for VPC and VMware vSphere.



Table 2. Mapping of network concepts between Bare Metal Servers for VPC and VMware vShpere

 Bare metal server VMware vSphere 

 PCI interface Uplink of the bare metal server on a standard vSwitch or distributed vSwitch. 
 VLAN interface Virtual network adapter of the VMKernel or virtual machine 
 VLAN ID VLAN ID for port group 



The PCI interface of the bare metal server maps to the Uplink in vSphere. When you provision a bare metal server, a primary PCI interface is created by default. This primary PCI interface automatically becomes the bare metal server’s uplink on vSwitch0.