vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Specifically, this means the failover occurs without using the FORCE parameter to acknowledge warnings in code or in the SSMS dialog boxes. It is only possible to have a planned failover to a secondary replica in synchronouscommit availability mode. You can move an asynchronous-commit availability mode replica to synchronous, wait for the SYNCHRONIZED state, and then issue a planned failover without data loss. Planned failovers should always be initiated via SSMS, T-SQL, or PowerShell.
* Forced failover - A manually-initiated, forced failover should only be initiated in response to adverse cluster conditions such as the loss of the primary node. Initiate from SSMS wizards, T-SQL commands, or PowerShell and only from Windows Server Failover Cluster Manager as a last resort.
* Force failover if WSFC quorum is down - You will not be able to force a failover for availability groups based on a WSFC if the WSFC has no quorum. You will first have to force quorum in the Configuration Manager by “rigging” the vote and modifying node weights. You should consider this step only in emergencies, such as when a disaster has disrupted a majority of cluster nodes. This can be accomplished this with a PowerShell script, to force an online node to assume the primary role without a majority of votes.



Failovers are not caused by database issues such as a database becoming suspect due to a loss of a data file or corruption of a transaction log.





 Other types of availability groups 

The following availability groups are not used in this deployment pattern, but may be of interest to solutioners wanting to adapt or extend this deployment pattern:



* Basic availability groups - A limited version of availability groups and are supported only on SQL Server Standard edition. The single secondary replica cannot be readable or backed up and each basic availability group can support only two replicas, but multiple basic availability groups can be configured per server. Basic availability groups allow for many of the same features of availability groups including synchronous or asynchronous replication, manual or automatic failovers.
* Distributed availability groups - This allows an availability group to treat another availability group to act as a secondary replica.
Started cl-sap-1
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-1
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-1
* Clone Set: fs_sapmnt-clone [fs_sapmnt]:
* Started: [ cl-sap-1 cl-sap-2 ]

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled







 Test2 - Testing failure of the node that is running the ASCS instance 

Use the folling information to test a failure of the node that is running the ASCS instance.



 Test2 - Description 

Simulate a crash of the node where the ASCS instance is running.





 Test2 - Prerequisites 



* A functional two-node RHEL HA Add-On cluster for SAP ENSA2.
* Both cluster nodes are active.
* Cluster is started on NODE1 and NODE2.



* Resource group ${sid}_ascs${ASCS_nr}_group is active on NODE2.
* Resources ${sid}_vip_ascs${ASCS_nr}, ${sid}_fs_ascs${ASCS_nr}_lvm, ${sid}_fs_ascs${ASCS_nr} and ${sid}_ascs${ASCS_nr} are Started on NODE2.
* Resource group ${sid}_ers${ERS_nr}_group is active on NODE1.
* Resources ${sid}_vip_ers${ERS_nr}, ${sid}_fs_ers${ERS_nr}_lvm, ${sid}_fs_ers${ERS_nr} and ${sid}_ers${ERS_nr} are Started on NODE1.



* Check SAP instance processes:



* ASCS instance is running on NODE2.
* ERS instance is running on NODE1.
* Resources ${sid}_vip_ascs${ASCS_nr}, ${sid}_fs_ascs${ASCS_nr}_lvm, ${sid}_fs_ascs${ASCS_nr} and ${sid}_ascs${ASCS_nr} are Started on NODE1.
* Resource group ${sid}_ers${ERS_nr}_group is active on NODE2.
* Resources ${sid}_vip_ers${ERS_nr}, ${sid}_fs_ers${ERS_nr}_lvm, ${sid}_fs_ers${ERS_nr} and ${sid}_ers${ERS_nr} are Started on NODE2.



* Check SAP instance processes:



* ASCS instance is running on NODE1.
* ERS instance is running on NODE2.









 Test4 - Test Procedure 

Log in to NODE1 and run sapcontrol to move the ASCS instance to the other node.

sudo -i -u ${sid}adm -- sh -c "sapcontrol -nr ${ASCS_nr} -function HAFailoverToNode"





 Test4 - Expected behavior 



* sapcontrol interacts with the cluster through the sap-cluster-connector.
* The cluster creates location constraints to move the resource.



Check the status with the following command. Keep in mind that the ASCS resource group moved to the second node. If you run the pcs status command too soon, you might see some resources stopping and starting.

pcs status

Sample output:

 pcs status
Cluster name: SAP_ASCS
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 09:03:19 2023
* Last change: Tue Feb 14 09:01:40 2023 by s01adm via crm_resource on cl-sap-1
Calling an action from another action 

In this tutorial, learn how to call an action from another action by creating a custom sequence in IBM Cloud® Functions. This tutorial uses Node.js 10, but the general principles can be applied to other runtimes.

This tutorial uses Cloud Functions to build a custom sequence within another Node.js action to model a typical scenario, such as getting data from IBM Cloud Object Storage, retrieving related data from IBM Cloudant database, and then putting the results back into IBM Cloud Object Storage. To accomplish this task, first create an IBM Cloud Object Storage action, then an IBM Cloudant database action, and finally a main action that calls both of them.

Note that this tutorial is a simulation; it does not connect to IBM Cloud Object Storage and IBM Cloudant. This tutorial does, however, run successfully to show you how to call an action from another action. You can use this tutorial as a building block to create something new. Because access to both IBM Cloud Object Storage and IBM Cloudant is asynchronous, to simulate asynchronous code, the actions use a timeout to complete these actions at a later point in time. This simulation is important to verify whether the custom sequence action code, which uses the openwhisk SDK, is using promises properly.

The code for the actions also includes some basic error handling code, including explanations. Note that, when you run these actions, you might not ever see the error handling code in this example. However, you might find this code useful if you choose to adapt the code examples.

Before you begin

Select a namespace to contain your Cloud Functions entities from the [console](https://cloud.ibm.com/functions). For more information about namespaces, see [Managing namespaces](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-namespaces).



 Step 1: Create the cos-access actioncos-access action 

The cos-access action is a Node.js program that simulates code to access IBM Cloud Object Storage by setting a timeout that resolves after 10 seconds.



1.
High-Availability 

IBM Cloud® Messages for RabbitMQ is a managed cloud messages service that is fully integrated into the IBM Cloud environments. The message broker, storage, and supporting infrastructure all run in IBM Cloud.



 RabbitMQ Cluster Configuration 

Messages for RabbitMQ provides replication, fail-over, and high-availability features to protect your databases and data from infrastructure maintenance, upgrades, and failures. Deployments contain a cluster with three nodes where users, virtual hosts, queues, exchanges, bindings, runtime parameters, and other distributed state are shared across all three nodes. If a node fails, the node is either deleted or restarted, and then it or a new node is resynced to the cluster. Your deployment remains available to process messages during the resync, although it is a memory-intensive process. For more information, see the [Performance](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=messages-for-rabbitmq-performance) page.





 RabbitMQ Resource Considerations 

As a message broker implemented in Erlang, RabbitMQ is thoroughly dependent on CPU cores to schedule its processes. If too few cores are available to run these processes, performance can suffer. If too few CPU resources are available to process them, RabbitMQ closes connections. Consider [cpu-contention](https://www.rabbitmq.com/runtime.htmlcpu-contention) and choose a resource configuration that provides enough CPU cores for RabbitMQ running as a production system. Having dedicated CPU resources in your RabbitMQ deployment also prevents the impact of noisy neighbors on resources, such as CPU and memory.





 High-Availability Queue Configuration 

The default virtual host is configured to mirror its queues across all nodes in the cluster to provide [high-availability](https://www.rabbitmq.com/ha.html). High availability in RabbitMQ is set by [policy](https://www.rabbitmq.com/parameters.htmlpolicies).
So when your action completes, the container "freezes" to pause all processes and then "wakes" when your action is rerun.

This approach affects garbage collection. When you run a Node.js action, garbage is created on the heap. This garbage is accumulating over each warm reuse of the action process. However, because the Node.js process pauses between invocations, garbage collection does not run reliably.

 How to fix it 

Invoke Node.js garbage collection explicitly from within your action code by adding the option --expose-gc.

For example, use code similar to the following example code in your action:

try {
if (global.gc) {
console.log("About to run garbage collection.");
global.gc();
console.log("Completed running garbage collection.");
} else {
console.log("Garbage collection not available.");
}
} catch (e) {
console.log("Garbage collection cannot be started: " + e);
}
Dual AZ deployment pattern 

IBM Cloud Windows Server 2019 Standard virtual servers configured as Windows Server Failover Cluster (WSFC) nodes, and SQL Server Enterprise edition with SQL Server Always On availability groups configured on each node form the basis of the dual AZ deployment pattern. This architecture along with MS ADDNS and a Windows file share used as a witness server provides a highly available SQL Server deployment.

Zoom

![Dual AZ deployment pattern](https://cloud.ibm.com/docs-content/v1/content/0144bdaedd7cbff0fe5c7d04bb02079a0fb9e634/microsoft//images/dualaz.svg)

Figure 1. Dual AZ deployment pattern

The dual AZ deployment pattern shown is suitable for production databases that require high availability and leverages the following core technologies:



* An IBM Cloud VPC with subnets in three AZs in the same MZR.
* Security groups are used to control traffic flows between the components.
* One or more bastion hosts used for administrative access to the servers.
* A Floating IP (FIP) attached to the bastion hosts to allow Internet access.
* Two Windows Server 2019 Standard virtual servers, one in AZ1 and AZ2, that are Active Directory domain controllers in the same forest\domain.
* Two Windows Server 2019 Standard virtual servers, one in AZ1 and AZ2, that will become Windows Server Failover Cluster (WSFC) nodes.
* A Windows 2019 Standard virtual server in the third AZ to provide a file share to enable cluster quorum.
* Windows Server Failover Cluster is the underlying clustering technology for the server resources and is a prerequisite for high availability Always On availability groups on Windows servers.
* Always On availability groups provide the ability to keep a discrete set of databases highly available across one or more cluster nodes and work at the database level. Availability groups consist of one primary replica and up to a maximum of eight secondary replicas, and use synchronous or asynchronous data replication. In this deployment synchronous replication is used due to the low latency between the AZs.
* Distributed Network Names is a name resource in WSFC and Always On availability groups, used for name resolution of the cluster resources.
The features do include manual planned and forced failover and both synchronous and asynchronous replication modes, readable secondary nodes, and secondary replica backups. Availability groups without a WSFC cluster can still provide readable secondary nodes, read-only routing, and load balancing. Failover can be automated with external tools. A new feature of SQL Server 2019 helps mitigate this with automatic traffic redirection, which provides for routing of read/write traffic back to the primary node and does not require a Listener.



The following terminology is used to describe availability group concepts:



* Availability group - supports a replicated environment for a discrete set of user databases.
* HA availability group - a group of databases that fail over together.
* Read-scale availability group - a group of databases that are copied to other instances of SQL Server for read-only workload.
* Primary replica - makes the primary databases available for read-write connections from clients and sends transaction log records of each primary database to every secondary database.
* Secondary replica - up to eight secondary replicas, each of which hosts a set of secondary databases and serves as a potential failover targets for the HA availability group.





 Synchronization modes 

Availability group secondary replicas can be configured with one of the following synchronization modes:



* Synchronous - The log is hardened (the transactions are committed to the transaction log) on every secondary replica before the transaction is committed on the primary replica. This guarantees zero data loss, with a potential performance impact on a highly transactional workload if network latency is high. You can have two synchronous-commit replicas per AG. This mode is best suited for instances in the same AZ or MZR.
* Asynchronous - The transaction is considered committed as soon as it is hardened in the transaction log on the primary replica. If there is an issue before the logs are hardened on all of the secondary replicas, data loss is possible, and the recovery point would be the most recently committed transaction that was successful on all of the secondary replicas. This mode is better suited for instances in different MZRs or between an AZ and on-premise.







 Failover modes
Dual multi-zone region deployment pattern 

IBM Cloud Windows Server 2019 Standard virtual servers configured as Windows Server Failover Cluster (WSFC) nodes, and SQL Server Enterprise edition with SQL Server Always On availability groups configured on each node form the basis of the dual Multi-Zone Region (MZR) deployment pattern. This architecture along with MS ADDNS provides a SQL Server deployment for disaster recovery.

Zoom

![Dual MZR deployment pattern](https://cloud.ibm.com/docs-content/v1/content/0144bdaedd7cbff0fe5c7d04bb02079a0fb9e634/microsoft//images/dualmzr.svg)

Figure 1. Dual MZR deployment pattern

The dual MZR deployment pattern shown leverages the dual Availability Zone (AZ) pattern, and extends it so that there is a copy of the databases in a remote region; therefore, this pattern is suitable for production databases that require disaster recovery and leverages the following core technologies:



* An IBM Cloud VPC with subnets in two MZRs.
* Security groups are used to control traffic flows between the components.
* One or more bastion hosts used for administrative access to the servers.
* Floating IP (FIP) attached to the bastion hosts to allow Internet access.
* Three Windows Server 2019 Standard virtual servers, one in each AZ in the primary MZR and a third in the recovery MZR, that are Active Directory domain controllers in the same forest\domain.
* Three Windows Server 2019 Standard virtual servers, one in each AZ in the primary MZR and a third in the recovery MZR, that will become Windows Server Failover Cluster (WSFC) nodes.
* Always On availability groups provide the ability to keep a discrete set of databases highly available across one or more cluster nodes and work at the database level. Availability groups consist of one primary replica and up to a maximum of eight secondary replicas, and use synchronous or asynchronous data replication. In this deployment:



* synchronous replication ios used between the two AZs in the primary MZR.
* asynchronous replication is used between the MZRs.
pacemaker: active/disabled
pcsd: active/enabled





 Test1 - Test Procedure 

Crash SAP ASCS instance by sending a SIGKILL signal as user ${sid}adm to the enque server.

On NODE1, identify the PID of the enque server.

pgrep -af "(en|enq).sap"

Send a SIGKILL signal to the identified process.

Sample output:

 pgrep -af "(en|enq).sap"
30186 en.sapS01_ASCS01 pf=/usr/sap/S01/SYS/profile/S01_ASCS01_cl-sap-scs
 kill -9 30186





 Test1 - Expected behavior 



* SAP ASCS instance on NODE1 crashes.
* The cluster detects the crashed ASCS instance.
* The cluster stops the dependent resources on NODE1 (virtual IP address, file system /usr/sap/${SID}/ASCS${ASCS_nr}, and the LVM resources), and acquires them on NODE2.
* The cluster starts the ASCS on NODE2.
* The cluster stops the ERS instance on NODE2.
* The cluster stops the dependent resources on NODE1 (virtual IP address, file system /usr/sap/${SID}/ERS${ERS_nr}, and the LVM resources), and acquires them on NODE1.
* The cluster starts the ERS on NODE1.



After a few seconds, check the status with the following command.

pcs status

Sample output:

 pcs status
Cluster name: SAP_ASCS
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 08:10:18 2023
* Last change: Tue Feb 14 05:02:22 2023 by hacluster via crmd on cl-sap-1
* 2 nodes configured
* 11 resource instances configured

Node List:
* s01_fs_ers02_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-1
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-1
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-1
* Clone Set: fs_sapmnt-clone [fs_sapmnt]:
* Started: [ cl-sap-1 cl-sap-2 ]

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled





 Test4 - Recovery Procedure 

Wait until the ASCS instance is active on NODE2. After five minutes, the cluster removes the created location constraints automatically.

The following instructions show how to remove the constraints manually.

On NODE2, run the following command.

pcs constraint

Sample output:

 pcs constraint
Location Constraints:
Resource: s01_ascs01_group
Constraint: cli-ban-s01_ascs01_group-on-cl-sap-1
Rule: boolean-op=and score=-INFINITY
Expression: uname eq string cl-sap-1
Expression: date lt 2023-02-08 09:33:50 -05:00
Ordering Constraints:
start s01_ascs01_group then stop s01_ers02_group (kind:Optional) (non-symmetrical)
start fs_sapmnt-clone then start s01_ascs01_group (kind:Mandatory)
start fs_sapmnt-clone then start s01_ers02_group (kind:Mandatory)
Colocation Constraints:
s01_ers02_group with s01_ascs01_group (score:-5000)
Ticket Constraints:

pcs resource clear ${sid}_ascs${ASCS_nr}_group

The Location constraints are removed:

pcs constraint

Sample output:
* Last updated: Tue Feb 14 08:41:23 2023
* Last change: Tue Feb 14 08:34:04 2023 by hacluster via crmd on cl-sap-1
* 2 nodes configured
* 11 resource instances configured

Node List:
* Online: [ cl-sap-1 cl-sap-2 ]

Full List of Resources:
* res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-sap-1
* Resource Group: s01_ascs01_group:
* s01_vip_ascs01	(ocf::heartbeat:IPaddr2):	 Started cl-sap-1
* s01_fs_ascs01_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-1
* s01_fs_ascs01	(ocf::heartbeat:Filesystem):	 Started cl-sap-1
* s01_ascs01	(ocf::heartbeat:SAPInstance):	 Started cl-sap-1
* Resource Group: s01_ers02_group:
* s01_vip_ers02	(ocf::heartbeat:IPaddr2):	 Started cl-sap-2
* s01_fs_ers02_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-2
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-2
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-2
* Clone Set: fs_sapmnt-clone [fs_sapmnt]:
* Started: [ cl-sap-1 cl-sap-2 ]

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
vzcheckovr Utility to check the current system overcommitment and safety of the total resource control settings. 
 vzstat Utility to monitor the hardware node and container resources consumption in real time. 
 vzpid Utility that prints container ID that the process belongs to. 
 vzsplit Utility to generate container configuration file sample, “splitting” the hardware node into equal parts. 
 vzcfgscale Utility to scale the container configuration. 
 vzcfgvalidate Utility to validate container configuration file correctness. 
 vzcfgconvert Utility to convert Virtuozzo 2.0.2 container configuration files to Virtuozzo 2.5.x format. 
 vzstatrep Utility to analyze the logs that are collected by vzlmond and to generate statistics reports based on these logs (in the text and graphical form). 
 vzreport Utility to draw up a problem report and to automatically send it to the Parallels support team. 
 vzhwcalc Utility to scan the main resources on any Linux server and create a file where this information is specified. 
 vzveconvert Utility to convert the containers based on Virtuozzo standard OS templates to the EZ template-based ones. 
 vznetcfg Utility to manage network devices on the hardware node. 
 vzmtemplate Utility to migrate the installed OS and application templates from the one hardware node to another. 







 Common Terms 



Table 7. Common terms

 Common terms 

 Hardware Node Hardware node (or HN) is a physical computer that hosts containers. 
 Container Container is functionally identical to an isolated stand-alone server with its own IP addresses, processes, files, users, its own version, its own configuration files, its own applications, system libraries, and other server functions. Containers share hardware nodes and the same OS kernel, but they are isolated from each other. Each container has a unique numerical ID starting with 1. 
 Host Operating System Host operating system (or Host) is an operating system that is installed on the hardware node. It is also called container 0, in contrast to real containers that have IDs that start with 1.
In the following example output, the node port is 31024:

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
webserver-lb 172.21.xxx.xxx 169.xx.xxx.xxx 80:31024/TCP 2m run=webserver
2. For classic clusters, get the Public IP address of a worker node. For VPC clusters, get the Private IP address instead.

ibmcloud ks worker ls --cluster <cluster_name>

Example output

ID Public IP Private IP Machine Type State Status Zone Version
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w1 169.xx.xxx.xxx 10.176.48.67 u3c.2x4.encrypted normal Ready dal10 1.26_1513
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w2 169.xx.xxx.xxx 10.176.48.79 u3c.2x4.encrypted normal Ready dal10 1.26_1513
kube-dal10-cr18e61e63c6e94b658596ca93d087eed9-w3 169.xx.xxx.xxx 10.176.48.78 u3c.2x4.encrypted normal Ready dal10 1.26_1513
3. Copy the public IP of the worker node and the node port into your text cheat sheet to use in later lessons.
4. Verify that you can access the public IP address the worker node through the node port. Note: Because worker nodes in VPC clusters don't have a public IP address, you can access an app through a NodePort only if you are connected to your private VPC network, such as through a VPN connection. Then, you can use the worker node's private IP address and NodePort: <worker_private_IP>:<NodePort>.