vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* The buildpack also contains updated versions of IBM JRE: 8 SR2 FP12 and 7.1 SR3 FP32.
* The buildpack provides an updated version of the agent for the Auto-Scaling service.
* The buildpack now comes with a new data collector for the Monitoring and Analytics service. The new collector enables configuration of monitoring thresholds and contains a number of bug fixes.
* The buildpack provides an updated DB2® JDBC driver version 4.19.49.
* The Node.js runtime that is used by the [devconsole and shell App Management utilities](https://cloud.ibm.com/docs/cloud-foundry-public?topic=cloud-foundry-public-app_management) was updated to the latest 0.12.12 version.







 18 March 2016 

Updated Node.js buildpack v3.2-20160315-1257
: This release of the buildpack moves the default IBM SDK for Node.js runtime from version 4.3.0 to 4.3.2. It also includes IBM SDK for Node.js versions 0.10.43, 0.12.12, and 4.3.1. Use these recent Node.js versions to pick up fixes for several security vulnerabilities.





 7 March 2016 

Updated Liberty buildpack v2.6-20160225-1649
: The buildpack adds support for Dynatrace app monitoring. See [Using Dynatrace](https://cloud.ibm.com/docs/cloud-foundry-public?topic=cloud-foundry-public-using_dynatrace) for details.



* The buildpack adds support for DynamicPULSE.







 4 March 2016 

Updated Node.js buildpack v3.1-20160222-1123
: This release of the buildpack moves the default IBM SDK for Node.js runtime from version 4.2.4 to 4.3.0. It also includes IBM SDK for Node.js versions 0.10.42, 0.12.10, and 4.2.6.
Network topology 

There are many ways to connect to IBM Cloud® Object Storage and the choice of endpoint can have an impact on performance.



 Physical distance 

When an application makes a request to COS, it needs to cross some amount of physical distance. As this distance increases, the latency of the request will also increase. In order to lessen the latency imposed by physical distance, it is optimal to co-locate compute resources and object storage where possible. If your application is running in the IBM Cloud in the us-south region, then in order to optimize performance it would be best to read and write data to a bucket also located in the us-south region.

Workloads that require accessing data in far reaching places might benefit from using IBM Aspera, especially if there is significant packet loss. More information about using IBM Aspera High-Speed Transfer and COS can be found in the Aspera guide.

Applications with global reach will benefit from using a Content Delivery Network to cache assets stored in COS in locations closer to their end users. The original files continue to be hosted in their bucket, but copies can be cached in various locations around the world where users are originating requests.





 Resilience requirements 

Some workloads might require the additional levels of resiliency that comes with writing data to Cross Region buckets, while others might rely on the increased marginal performance found in a Single Data Center bucket. Each application needs to strike a balance between higher availability and faster performance.

When using a Cross Region endpoint, it is possible to direct inbound traffic to a specific access point while still dispersing data across all three regions. When sending requests to an individual access point there is no automated failover if that region becomes unavailable. Applications that direct traffic to an access point instead of the geo endpoint must implement appropriate failover logic internally to achieve the availability advantages of the cross-region storage.

One reason for using an access point is to control where data ingress and egress occurs while still dispersing the data across the widest possible area. Imagine an application running in the us-south region that wants to store data in a US cross-region bucket but wants to ensure that all read and write requests remain in the Dallas area:



1.
This scenario causes problems when upgrading from a pre-20.4R2-S2 version to 20.4R2-S2, because the commit of the previous configuration to the new release fails, causing the upgrade to fail as well.





 Command set security datapath-debug.. 

set security datapath-debug configuration commands are known to cause errors during when upgrading. You should remove all datapath-debug commands from the config and retry the readiness check. For example, commands like these should be removed:

set security datapath-debug capture-file pcap001
set security datapath-debug capture-file format pcap
set security datapath-debug capture-file size 10m
set security datapath-debug capture-file files 5
set security datapath-debug maximum-capture-size 1500







 Correcting warning 1176 

A VPN configuration with establish-tunnels not set to immediately was detected. After an upgrade, the IKE might not be immediately active depending on negotiations with the remote peer gateway, and whether or not data traffic is actively flowing. Without establish-tunnels immediately, the tunnel is established with on-traffic. With the establish-tunnels immediately statement, the tunnel is established immediately when the configuration is committed. However, establish-tunnels immediately might trigger an undesirable outcome when configured on both ends of the tunnel.

For more information on this issue, see [(SRX) IPSec comes UP when SRX-A is the Initiator, but fails when SRX-A becomes the responder](https://kb.juniper.net/InfoCenter/index?page=content&id=KB22239) in the Juniper Knowledge Base. Consult [vpn (Security)](https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/security-edit-vpn.html) for more details on these settings.





 Correcting warning 1177 

One or more security zone policy rules with the dynamic-application any configuration was detected.
This pattern can also be used for other programming languages where a compilation needs to be performed.



* Node applications that require a build, for example an Angular or React build. Here, the builder and runtime base image might end up being the same (both node), but not all build time artifacts and sources need to be copied into the runtime image.
* Any programming language that compiles source code into a native executable that runs without a runtime environment, for example Go or Rust.



For those languages that produce a native executable and do not need a runtime environment at all, use another Docker capability for the runtime stage: scratch. Scratch can be used as a base in the FROM command, but is not a final container image. Instead, it tells the Docker build to not use a base image at all. Without any operating system files from a base image, the result image can contain as little as a single file: your binary that is copied over from the builder stage. Note that depending on the programming language and your code, you might have to make further adjustments on compiler options as binary files might rely on some operating system files to exist.





 Keep your image clean 

When you are first developing your Dockerfile and debugging how it works, you might install some temporary tools or apply other workloads that are not required to be in the final container image. Removing those temporary files and workloads keeps your image clean, small, and more secure.







 Improving the start time of your image 

For maximum efficiency, your container image must start as fast as possible. For applications, the relevant metric is how long it takes for the HTTP endpoint to be available and ready to accept and process incoming HTTP requests. Start speed is especially important for Code Engine applications that are based on Knative. Such applications can be configured to scale down to zero running instances when there is no traffic, thus consuming no resources and costing no money. When a request comes in, an instance is started to handle the request. Your container must start as fast as possible to respond to the new request.

To improve the startup of your application, investigate the implementation of your application and see whether you can apply patterns such as:
Meaning it can include even skills and assistants to which you do not have access.
* The access permissions information of the original service instances is not stored in the backup. Meaning original access rights, which determine who can see a service instance and who cannot, are not preserved.
* You cannot use this procedure to back up the data that is returned by the search skill. Data that is retrieved by the search skill comes from a data collection in a Discovery instance. See the [Discovery documentation](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-backup-restore) to find out how to back up its data.
* If you back up and restore or otherwise change the Discovery service that your search skill connects to, then you cannot restore the search skill, but must recreate it. When you set up a search skill, you map sections of the assistant's response to fields in a data collection that is hosted by an instance of Discovery on the same cluster. If the Discovery instance changes, your mapping to it is broken. If your Discovery service does not change, then the search skill can continue to connect to the data collection.
* The tool that restores the data clears the current database before it restores the backup. Therefore, if you might need to revert to the current database, create a backup of it first.
* The target IBM Cloud Pak for Data cluster where you restore the data must have the same number of provisioned Watson Assistant service instances as the environment from which you back up the database. To verify in the IBM Cloud Pak for Data web client, select Services from the main navigation menu, select Instances, and then open the Provisioned instances tab. If more than one user created instances, then ask the other users who created instances to log in and check the number they created. You can then add up the total sum of instances for your deployment. Note that not even an administrative user can see instances that were created by others from the web client user interface.







 Backing up data by using the CronJob 

A CronJob named $INSTANCE-store-cronjob is created and enabled for you automatically when you deploy the service. A CronJob is a type of Kubernetes controller.
IBMSSEKPCustomerRootKeyCrn=COS_KP_ROOTKEY_CRN
)
print("Encrypted Bucket: {0} created!".format(bucket_name))
except ClientError as be:
print("CLIENT ERROR: {0}n".format(be))
except Exception as e:
print("Unable to create encrypted bucket: {0}".format(e))
Show more

Key Values



* <algorithm> - The encryption algorithm that is used for new objects added to the bucket (Default is AES256).
* <root-key-crn> - CRN of the Root Key that is obtained from the Key Protect service.



SDK References



* Classes



* [Bucket](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlbucket)



* Methods



* [create](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlS3.Bucket.create)











 Using Aspera High-Speed Transfer 

By installing the [Aspera high-speed transfer library](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-asperaaspera-packaging), you can use high-speed file transfers within your application. The Aspera library is closed-source, and thus an optional dependency for the COS SDK (which uses an Apache license).

Each Aspera session creates an individual ascp process that runs on the client machine to perform the transfer. Ensure that your computing environment can allow this process to run.



 Initializing the AsperaTransferManager 

Before initializing the AsperaTransferManager, make sure that you have a working [client](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlclient) (not a resource or session) object.

import ibm_boto3
For PITR, use the point_in_time_recovery_time and point_in_time_recovery_deployment_id parameters. The point_in_time_recovery_deployment_id is the source deployment's ID and point_in_time_recovery_time is the timestamp in UTC you want to restore to. If you want to restore to the latest available point-in-time use "point_in_time_recovery_time":" ".

If you need to adjust resources or use a Key Protect key, add the optional parameters key_protect_key, members_disk_allocation_mb, members_memory_allocation_mb, and/or members_cpu_allocation_count, and their values to the body of the request.







 Verifying PITR 

To verify the correct recovery time, check the database logs. Checking the database logs requires the [Logging Integration](https://cloud.ibm.com/docs/databases-for-mysql?topic=cloud-databases-logging) to be set up on your deployment.

When you perform a recovery, your data is restored from the most recent incremental backup. Any outstanding transactions from the WAL log are used to catch your database up to the time you recovered to. After the recovery is finished, and the transactions are run, the logs display a message. You can check that your logs have the message,

LOG: last completed transaction was at log time 2019-09-03 19:40:48.997696+00

There are two scenarios where recovery does not show up in the logs.



1. Your deployment has a recent full backup and there is no activity after the backup was taken that needs to be replayed.
2. If you entered a time to recover to that is after the current time or is past latest available point-in-time recovery point.



In both cases the recovery is usually still successful, but there won't be an entry in the logs to check the exact time that the database was restored to.
Disabling Intel Hyper-Threading Technology 

If your infrastructure configuration and application setup requires it, you can disable Intel Hyper-Threading Technology (Intel HT Technology) on your guest virtual machine.

The instructions in this topic for disabling Intel Hyper-Threading Technology on guest VMs apply only to virtual servers that are provisioned on hosts that use Intel-based processors.



 Understanding reasons to disable Intel Hyper-Threading Technology 

Intel Hyper-Threading Technology is a term that describes simultaneous multithreading (SMT). Hyper-Threading Technology splits each physical core into two virtual processors. Hyper-Threading Technology is like taking a wide road with a single lane and making it into two relatively narrower lanes. The two-lane highway provides better service over the single lane road if there is slow and fast moving traffic. Hyper-Threading Technology provides better application performance when there is File I/O, Network I/O and other slower operations mixed with CPU intensive operations. The performance advantage of Hyper-Threading Technology typically ranges from 0 - 30% over a single-thread mode. Some applications might also see a drop in performance.

Hyper-Threading Technology is not always desirable. In this case, you might prefer to disable it and run the cores in single-threaded mode. Three main reasons exist to consider disabling Hyper-Threading Technology. The first reason is related to software licensing. Many legacy software vendors such as electronic design automation (EDA) tool providers, require a license for each CPU, regardless of whether they use Hyper-Threading Technology or single-threaded mode. When the core is in Hyper-Threading Technology mode, you need twice as many licenses as you do in single-threaded mode, even though the application is not getting two times better performance over single-threaded mode. As a result, EDA and other practitioners disable Hyper-Threading Technology and prefer to run their software in single-threaded mode.

The second reason for disabling Hyper-Threading Technology is to avoid load balancing issues and operating system overhead that comes with Hyper-Threading Technology.
DNS records](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap/images/sap-terraform-ha-dns.png)

Figure 6. DNS records







 2. Highly available system for SAP HANA database 

Zoom

![Figure 7. SAP HA for HANA DB instances cluster nodes Primary (Active) and Secondary (Passive)](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap/images/sap-ha-hana-vpc-single-zone.svg)

Figure 2. SAP HA for HANA DB instances cluster nodes Primary (Active) and Secondary (Passive)

At the most basic level, a standard HA HANA cluster in an active-passive configuration has two nodes: one is the primary node and the other is the standby node. This simply means that the primary node is actively serving the active SAP instances (PAS and AAS), while the standby node is waiting to jump in if there is a failure.

The cluster is set with a virtual hostname IP (hostname is mapped to the FQDN of the HANA ALB through DNS, which is the same as explained previously for SAP ASCS and ERS instances). App instances (PAS and AAS), these are the details to be used on the SAP profiles to call that particular component. The cluster assigns that virtual IP to the active node and uses a heartbeat monitor to confirm the availability of the components. If the primary node stops responding, it triggers the automatic failover mechanism that calls the standby node to step up to become the primary node. The ALB detects the change, redirects the traffic to the new active node, and assigns the virtual IP to it, restoring the component availability. After the failed node is fixed, it comes online as a standby node.



 Synchronous on disk (sync) HANA database replication mechanism supported by SAP
Understanding high availability and disaster recovery for Schematics 

Understand the high availability features of the IBM Cloud® resources that you want to provision with IBM Cloud® Schematics and design your resources to meet the availability requirements that your business and clients need.

High availability is a core discipline in an IT infrastructure to keep your resources healthy and your app workloads up and running, even after a partial or full site failure. The main purpose of high availability is to eliminate potential points of failure in an IT infrastructure. For example, you can prepare for the failure of one system by adding redundancy and setting up failover mechanisms.

How is IBM Cloud Schematics set up to ensure high availability of the service?

Schematics is deployed as two highly available service instances in two separate geographical locations, the US and Europe. Within each geography, the service is deployed across two multizone regions, such as us-south and us-east in the US geography, and eu-de and eu-gb for the Europe geography. This setup ensures that the service is still available, even if one region within a geography fails. Data is not shared across geographical locations.

Who is responsible to set up high availability for my resources?

While IBM Cloud Schematics is responsible to ensure that your workspace information is available, backed up, and replicated across multiple regions so that information can be recovered after a failure, Schematics does not set up high availability for your IBM Cloud resources. Instead, you must understand the features that each resource offering provides to decide what level of availability is the right one for your needs. Then, you use IBM Cloud Schematics to provision and configure your IBM Cloud resources in a highly available setup.

How can I implement high availability for my resources?

Review the following image to find a general approach of how you can make your resource highly available and add resiliency to account for a site or region failure. The level of availability that is right for you depends on several factors, such as the high availability features that are available for your resource, your business requirements, the [Service Level Agreements](https://cloud.ibm.com/docs/overview?topic=overview-slas) that you have with your clients, and the money that you need to expense.
Backing up and restoring data 

You can back up and restore the data that is associated with your Watson Assistant deployment in IBM Cloud Pak for Data.

The primary data storage for Watson Assistant is a Postgres database. Your data, such as workspaces, assistants, and skills are stored in Postgres. Other internal data, such as trained models, can be recreated from the data in Postgres.

Choose one of the following ways to manage the back up of data:



* [Kubernetes CronJob](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-backupbackup-cronjob): Use the $INSTANCE-store-cronjob cron job that is provided for you.
* [backupPG.sh script](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-backupbackup-os): Use the backupPG.sh bash script.
* [pg_dump tool](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-backupbackup-cp4d): Run the pg_dump tool on each cluster directly. This is the most manual option, but also affords the most control over the process.



When you back up data with one of these procedures before you upgrade from one version to another, the workspace IDs of your skills are preserved, but the service instance IDs and credentials change.



 Before you begin 



* When you create a backup by using this procedure, the backup includes all of the assistants and skills from all of the service instances. Meaning it can include even skills and assistants to which you do not have access.
* The access permissions information of the original service instances is not stored in the backup. Meaning original access rights, which determine who can see a service instance and who cannot, are not preserved.
* You cannot use this procedure to back up the data that is returned by the search skill. Data that is retrieved by the search skill comes from a data collection in a Discovery instance.
* You need an SSH key to connect to the virtual servers. If you don't have an SSH key, see [the instructions](https://cloud.ibm.com/docs/vpc?topic=vpc-ssh-keys) for creating a key for VPC.







 Step 1: Create a Virtual Private Cloud and subnets 

To tighten the security of your servers, it is recommended to only allow connections to the ports required by the applications deployed on the servers. In this tutorial, the application will be a web server, thus it will only need to allow inbound connections on port 80.

To perform maintenance tasks on these servers such as installing software, performing operating system upgrades, you will go through a bastion host. A bastion host is an instance that is provisioned with a public IP address and can be accessed via SSH. Once set up, the bastion host acts as a jump server allowing a secure connection to instances provisioned in the VPC.

In this section, you will create the VPC and the bastion host.

This tutorial also comes with companion shell scripts and a Terraform template, that can be used to generate the resources that you will create using the UI below. They are available [in this Github repository](https://github.com/IBM-Cloud/vpc-tutorials/tree/master/vpc-public-app-private-backend).



1. Navigate to the [Virtual Private Clouds](https://cloud.ibm.com/vpc-ext/network/vpcs) page and click on Create.
2. Under New Virtual Private Cloud section:



1. Enter vpc-pubpriv as name for your VPC.
2. Select a Resource group.
3. Optionally, add Tags to organize your resources.



3. Uncheck Allow SSH and Allow ping from the Default security group. SSH access will later be added to the maintenance security group. The maintenance security group must be added to an instance to allow SSH access from the bastion server. Ping access is not required for this tutorial.
4. Under Subnets change the name of the Zone 1 subnet.