vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For this example, we assume one backup every 24 hours and retain seven backups.

For the peer pods:



* CouchDB snapshot daily at 3:00 a.m.
* Peer snapshot daily at 3:05 a.m.



For the ordering node pods:



* Ordering node snapshots daily at 5:00 a.m. The two-hour gap allows sufficient time for all of the peers to be snapshotted from every organization before ordering nodes are snapshotted.



If you are using private data collections in your applications, your nodes must be running Hyperledger Fabric 2.2.1 or higher in order to back up peer and ordering nodes while they are running. When restoring from a backup, the peers must be restored from a lower block height than the ordering nodes. While peers can catch up to the block height of the ordering service, the ordering service does not have the actual private data (in a private data transaction, only hashes of the data are committed to the public ledger), rendering the data for these blocks unavailable. Fabric v2.2.1 and higher has a configurable option to lower the priority for the reconciliation of missing private data. Earlier versions constantly attempt to reconcile the missing private data, potentially blocking out reconciliation of other private data. On earlier versions of Hyperledger Fabric, you must back up ordering nodes and peers at exactly the same block heights to avoid this issue. This can be accomplished by scaling the both the peer and ordering node pods to 0 before taking the backup.







 Taking snapshots 

The overall steps for the backup and restore process apply in all cases and are independent of where your components are deployed. However, the specific commands that are used in this example for creating, listing, and restoring snapshots apply only if you are running on IBM Cloud. For other cloud providers or when running in your own datacenter, follow the specific instructions for underlying snapshot creation and management found in the documentation of those providers.

When a snapshot is being taken the node continues to function as normal.



 Node snapshots 

Because the process for taking a snapshot is the same for all nodes, only one set of instructions is shown here.
However, when you decide between shared and dedicated nodes, you might want to check with your legal department to discuss the level of infrastructure isolation and compliance that your app environment requires.

Some classic worker node flavors are available for only one type of tenancy setup. For example, m3c VMs can be provisioned in a shared tenancy setup only.





 How does bandwidth allocation work for VPC Gen 2 worker nodes? 

Note that Red Hat OpenShift on IBM Cloud doesn't support adjusting the bandwidth profile that is associated with the underlying VPC instance profile.

When you provision a worker node on a VPC Gen 2 cluster, you get the default bandwidth allocation for that instance profile. For more information, see [Bandwidth profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-bandwidth-allocation-profiles).





 How does storage work for VMs? 

Every VM comes with an attached disk for storage of information that the VM needs to run, such as OS file system, container runtime, and the kubelet. Local storage on the worker node is for short-term processing only, and the storage disks are wiped when you delete, reload, replace, or update the worker node. For persistent storage solutions for your apps, see [Planning highly available persistent storage](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-plan). Additionally, classic and VPC infrastructure differ in the disk setup.



* Classic VMs: Classic VMs have two attached disks. The primary storage disk has 25 GB for the OS file system, and the auxiliary storage disk has 100 GB for data such as the container runtime and the kubelet. For reliability, the primary and auxiliary storage volumes are local disks instead of storage area networking (SAN). Reliability benefits include higher throughput when serializing bytes to the local disk and reduced file system degradation due to network failures. The auxiliary disk is encrypted by default.
* VPC compute VMs: VPC VMs have one primary disk that is a block storage volume that is attached via the network. The storage layer is not separated from the other networking layers, and both network and storage traffic are routed on the same network.
This tutorial is stand alone and it is not required to execute the steps in [part one](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-vpc-transit1). If you are not familiar with VPC, network IP layout and planning in the IBM Cloud, Transit Gateway, IBM Cloud® Direct Link or asymmetric routing consider reading through part one.

The hub and spoke model supports a number of different scenarios:



* The hub can be the repository for shared micro services used by spokes and enterprise.
* The hub can be a central point of traffic firewall-router and routing between enterprise and the cloud.
* The hub can monitor all or some of the traffic - spoke <-> spoke, spoke <-> transit, or spoke <-> enterprise.
* The hub can hold the VPN resources that are shared by the spokes.
* The hub can be the repository for shared cloud resources, like databases, accessed through [virtual private endpoint gateways](https://cloud.ibm.com/docs/vpc?topic=vpc-about-vpe) controlled with VPC security groups and subnet access control lists, shared by spokes and enterprise



There is a companion [GitHub repository](https://github.com/IBM-Cloud/vpc-transit) that divides the connectivity into a number of incremental layers. In the tutorial thin layers enable the introduction of bite size challenges and solutions.

The following will be explored:



* VPC egress and ingress routing.
* [Virtual Network Functions](https://cloud.ibm.com/docs/vpc?topic=vpc-about-vnf-ha) in combination with a Network Load Balancers to support a high availability and scalability.
* Virtual private endpoint gateways.
* DNS resolution.



A layered architecture will introduce resources and demonstrate connectivity. Each layer will add additional connectivity and resources. The layers are implemented in Terraform. It will be possible to change parameters, like number of zones, by changing a Terraform variable.
If the primary node stops responding, it triggers the automatic failover mechanism that calls the standby node to step up to become the primary node. The ALB detects the change, redirects the traffic to the new active node, and assigns the virtual IP to it, restoring the component availability. After the failed node is fixed, it comes online as a standby node.



 Synchronous on disk (sync) HANA database replication mechanism supported by SAP 

The primary system waits to commit the transaction until it gets a reply that the log is persisted in the secondary system. This option guarantees immediate consistency between both systems, at a cost of delaying the transaction by the time for data transmission and persisting in the secondary system.

When the connection to the secondary system is lost, the primary system continues the transaction processing and writes the changes only to the local disk. No data loss occurs in this scenario if the secondary system is connected. Data loss can occur when a takeover is run while the secondary system is disconnected.

Additionally, this replication mode can run with a full sync option. This means that log write is successful when the log buffer has been written to the log file of the primary and the secondary system. When the secondary system is disconnected (for example, because of network failure), the primary system suspends the transaction processing until the connection to the secondary system is reestablished. No data loss occurs in this scenario. You can set the full sync option for system replication with the parameter [system_replication]/enable_full_sync.

If SAP HANA system replication runs in the sync replication mode with the full sync option enabled, and if the connection to the secondary site is interrupted, no write operations on the primary site are possible. The operation of creating a tenant database, for example, waits until the connection to the secondary is reestablished or the SQL statement times out.

This automation is offered free of charge however, the provisioned infrastructure comes at cost.
* proactively pre-fetch contents from origin servers so that users can access these contents rapidly from the edge
* extend the duration of TCP connections for multiple requests
* automatically compress images for lower latency.





 Objectives 



* Deploy a sample dynamic web application to a Kubernetes Service cluster.
* Make static content globally available with IBM® Content Delivery Network.
* Enable the Dynamic Content Acceleration (DCA) capability for performance optimization of non-static content.



Zoom

![Architecture](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution52-cdn-dca/solution_52_architecture.svg)

Figure 1. Architecture diagram of the tutorial



1. The developer creates a simple dynamic application and produces a Docker container image.
2. The image is pushed to a namespace in Container Registry.
3. The application is deployed to Kubernetes Service.
4. User accesses the application.
5. The application is accelerated through the Dynamic Content Acceleration capability of IBM® Content Delivery Network.
6. IBM® Content Delivery Network interacts with the application to fetch dynamic contents.







 Before you begin 

This tutorial requires:



* IBM Cloud CLI,



* IBM Cloud Kubernetes Service plugin (kubernetes-service),
* Container Registry plugin (container-registry),
* dev plugin,



* a Docker engine,
* kubectl to interact with Kubernetes clusters,
* git to clone source code repository.



You will find instructions to download and install these tools for your operating environment in the [Getting started with solution tutorials](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-tutorials) guide.

In addition: A minimal cluster with one (1) zone, one (1) worker node and the smallest available size (Flavor) is sufficient for this tutorial.
Restricting network traffic to edge worker nodes 

Classic clusters

Edge worker nodes can improve the security of your IBM Cloud® Kubernetes Service cluster by allowing fewer worker nodes by isolating the networking workload.

When you mark these worker nodes for networking only, other workloads can't consume the CPU or memory of the worker node and interfere with networking.

If you want to restrict network traffic to edge worker nodes in a multizone cluster, you must have at least two edge worker nodes per zone for high availability of load balancer or Ingress pods. Create an edge node worker pool that spans all the zones in your cluster, with at least two worker nodes per zone.



 Isolating networking workloads to edge nodes 

Add the dedicated=edge label to worker nodes on each public or private VLAN in your cluster. The labels ensure that network load balancers (NLBs) and Ingress application load balancers (ALBs) are deployed to those worker nodes only. For NLBs, ensure that two or more worker nodes per zone are edge nodes. For ALBs, ensure that three or more worker nodes per zone are edge nodes. Both public and private NLBs and ALBs can deploy to edge worker nodes.

Before you begin



* Ensure that you have the following [IBM Cloud IAM roles](https://cloud.ibm.com/docs/containers?topic=containers-userschecking-perms):



* Any platform access role for the cluster
* Writer or Manager service access role for all namespaces



* [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)



To create an edge node worker pool,



1. [Create a worker pool](https://cloud.ibm.com/docs/containers?topic=containers-add_workersadd_pool) that spans all zones in your cluster and has at least two workers per zone if you use NLBs or 3 or more workers per zone if you use ALBs.
Figure 1 depicts the transaction flow on a Hyperledger Fabric blockchain network:

Zoom

![Transaction Flow](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/v10_txflow.svg)

Figure 1. Transaction flow on a Hyperledger Fabric network

On a Hyperledger Fabric network, the flow of data for queries and transactions is initiated by a client-side application by submitting a transaction request to a peer on a channel. The initial flow of data across the network is common to both queries and transactions:



1. Using APIs available in the SDK, a client application signs and submits a transaction proposal to the appropriate endorsing peers on the specified channel. This initial transaction proposal is a request for endorsement.
2. Each peer on the channel verifies the identity and authority of the submitting client, and (if valid) runs the specified chaincode against the supplied inputs. Based on the transaction results and the endorsement policy for the invoked chaincode, each peer returns a signed YES or NO response to the application. Each signed YES response is an endorsement of the transaction.

At this point in the transaction flow, the process diverges for queries and transactions. If the proposal called a query function in the chaincode, the application returns the data to the client. If the proposal called a function in the chaincode to update the ledger, the application continues with the following steps:
3. The application forwards the transaction, which includes the read/write set and endorsements, to the ordering service.
4. The transaction is then relayed to the ordering service. All channel peers validate each transaction in the block by applying the chaincode-specific Validation Policy and running a Concurrency Control Version Check.



* Any transaction that fail the validation process is marked as invalid in the block, and the block is appended to the channel's ledger.
* All valid transactions update the state database accordingly with the modified key/value pairs.
This setting allows worker nodes to communicate between zones.
* The user who links the service to the Kubernetes cluster must have Storage Manage permissions. From the IBM Cloud dashboard, navigate to Manage > Access (IAM), then choose Users, and click the user who will link the service to the Kubernetes cluster. Click the Classic infrastructure tab, expand the Services twistie, and select Storage Manage. Click Apply to add this permission for the user.





 Browsers 

The IBM Blockchain Platform console has been successfully tested on the following browsers:



* Chrome Version 111.0.5563.64 (Official Build) (x86_64)
* Safari Version 16.3 (18614.4.6.1.6)
* Firefox 110.0.1 (64-bit)







 Resources required 



 Cluster size recommendations 

When you link your IBM Blockchain Platform console to a Kubernetes cluster on IBM Cloud, you need to ensure that your Kubernetes cluster meets the minimum hardware resource requirements:



Table 1. Recommended cluster size on Kubernetes cluster on IBM Cloud

 Kubernetes cluster type Use case CPU RAM Worker nodes 

 Standard (Recommended) Suitable for MVPs 4 (Shared) 16 GB (Shared) multiple 
 Free** Suitable for evaluation 2 4 GB 1 



** Preview the IBM Blockchain Platform at no charge for 30 days when you link your IBM Blockchain Platform service instance to an IBM Cloud Kubernetes free cluster. Performance will be limited by throughput, storage and functionality. IBM Cloud will delete your Kubernetes cluster after 30 days and you cannot migrate any nodes or data from a free cluster to a paid cluster. Also note that when components are deployed to a free cluster, the resource allocation for the nodes is sized smaller than a paid cluster.

These resources are sufficient for testing and experimentation. The [Build a network tutorial](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-build-networkibp-console-build-network), in which you create two peers, two CAs, and a single node ordering service, takes up approximately 1.95 CPU.
Inter-node encryption 

For the current generation of plans hosted on AWS, inter-node encryption is always enabled. For the Flex and Flex Performance plans hosted on IBM Cloud, you can optionally choose to enable this form of encryption.

Inter-node encryption is an additional layer of security that protects inter-node traffic between internal Db2 nodes in a massively parallel processing (MPP) cluster. It supplements the SSL encryption that is already in use between the server and your applications.

Inter-node encryption ensures that your data is processed with encryption at all points from your application to physical storage and back, which protects your information from physical and software-based attacks. Leaving inter-node traffic unencrypted means that your data is transferred in plain text between the physical hardware nodes that make up your Db2 MPP instance. IBM already employs a variety of security protocols to protect such traffic to ensure the security and integrity of the unencrypted internal data flows. With this new feature, IBM provides an additional security layer to protect the data.

Due to the additional processing required to encrypt data on send and decrypt on receive, workloads with significant data movement may see a performance impact as little as 5% or as much as 20%. If your instance is hosted on IBM Cloud, your solution architect must evaluate this performance impact against your need to encrypt internal traffic flows. In some cases, it might be warranted (for instance in heavily regulated industries or where end users demand such encryption). In other cases, the existing in-depth security protocols may be sufficient.

IBM Cloud plans only: To enable inter-node encryption, open the web console, select Administration, and navigate to the Security -> Encryption tab. Enabling or disabling inter-node encryption can be done online, and takes effect immediately. Choosing to enable inter-node encryption does not impact the Service Level Agreement (SLA), self-service backups, database replication, or scaling.
For more information, see [Tags for Block Storage for VPC snapshots](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-about&interface=uisnapshots-about-tags).





 Can I use volume snapshot for disaster recovery? 

You can use your snapshots and backups to create volumes in case of an emergency. You can also create copies of your snapshot in other regions and use them to create volumes there. However, the snapshot and ackup services do not provide continual backup with automatic failover. Restoring a volume from a backup or snapshot is a manual operation that takes time. If you require a higher level of service for automatic disaster recovery, see IBM's [Cloud disaster recovery solutions](https://www.ibm.com/cloud/disaster-recovery).





 How many copies of my backup can I create in other regions? 

You can copy a snapshot from one region to another region, and later use that snapshot to restore a volume in the new region. Only one copy of the snapshot can exist in each region. You can't create a copy of the snapshot in the source (local) region.



* Can I add tags to a Block Storage for VPC snapshot?

 Can I add tags to a Block Storage for VPC snapshot? 

Depending on the action that you're performing, you can add user tags and access management tags to your snapshots. User tags are used by the backup service to periodically create backup snapshots of the volume. Access management tags help organize access to your Block Storage for VPC snapshots. For more information, see [Tags for Block Storage for VPC snapshots](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-about&interface=uisnapshots-about-tags).
* Can I use volume snapshot for disaster recovery?

 Can I use volume snapshot for disaster recovery? 

You can use your snapshots and backups to create volumes in case of an emergency.
[A separate channel with all of the nodes on the channel within a single country](https://cloud.ibm.com/docs/blockchain?topic=blockchain-console-icp-about-data-residencyconsole-icp-about-data-residency-use-case-channel)
4. [A separate channel with only ordering nodes from one country](https://cloud.ibm.com/docs/blockchain?topic=blockchain-console-icp-about-data-residencyconsole-icp-about-data-residency-raft)



Each approach provides an increased level of isolation and protection for your data, but requires additional effort to implement and manage. To help you understand how each option can be used to achieve data residency, we provide an overview of how data is shared within a Hyperledger Fabric network. We then present an example use case to illustrate how organizations within a blockchain consortium would use each option to separate their data and prevent it from leaving their region.



 How data is shared within an IBM Blockchain Platform network 

The architecture of Hyperledger Fabric that underlies the IBM Blockchain Platform is centered around three key components: an ordering service (made up of ordering nodes), Certificate Authorities (CA), and peers. Additionally, organizations send transactions to these nodes from client applications that use the [Fabric SDKs](https://hyperledger-fabric.readthedocs.io/en/release-2.2/getting_started.html). When considering data residency, it is important to understand how these components interact with and store data.

Peers host ledgers and smart contracts, with the latter governing how changes to the former are made. Ledgers themselves contain two distinct, though related, parts: a world state which records the current value of any key (such as an asset), and a blockchain which contains a literal “chain of blocks” that contain the records of each transaction. Peers receive state updates in the form of new blocks from the ordering service. They then use the blocks and the world state to confirm (or commit) transactions, update the world state and add the log of transactions on the blockchain.
* kubelet: The kubelet is a worker node agent that runs on every worker node and is responsible for monitoring the health of pods that run on the worker node and for watching the events that the API server sends. Based on the events, the kubelet creates or removes pods, ensures liveness and readiness probes, and reports back the status of the pods to the API server.
* vpn: The Konnectivity agent works with the Konnectivity server to securely connect the master to the worker node. This connection supports apiserver proxy calls to your pods and services, and oc exec, attach, and logs calls to the kubelet.
* Other components: The kube-system project also includes components to manage IBM-provided resources such as storage plug-ins for file and block storage, ingress application load balancer (ALB), and keepalived.



openshift-cloud-credential-operator
: The cloud credential operator manages a controller for Red Hat OpenShift components that request cloud provider credentials. The controller ensures that only the credentials that are required for the operation are used, and not any elevated permissions like admin. For more information, see the [GitHub project](https://github.com/openshift/cloud-credential-operator).

openshift-cluster-node-tuning-operator
: IBM manages the [node tuning operator](https://docs.openshift.com/container-platform/4.11/scalability_and_performance/using-node-tuning-operator.html), which runs a daemon set on each worker node in the cluster to tune worker nodes. You can apply custom profiles to your worker nodes to improve performance. For more information, see [Modifying worker node settings by using the Node Tuning Operator](https://cloud.ibm.com/docs/openshift?topic=openshift-kernelworker-nto).

openshift-cluster-samples-operator