vania-janet/multiturn-rag-retrieval-data huggingface dataset:
After you connect to your deployment, you see a basic overview. Included is a simple summary of the cluster and the default databases. The cluster contains three nodes, the two data nodes and the third arbiter node, so it shows the three hosts and their replica set. Also shown is the current MongoDB version. Databases for MongoDB Standard uses the Community version while Databases for MongoDB EE uses the Enterprise version of the MongoDB database.

Next, you see the default databases for your deployment, which all hold information related to the database instance. local holds replication data. config holds data for cluster operations. admin holds user authentication data. MongoDB Compass might not have access to all the data in these databases for permissions and security reasons.

Now you can use MongoDB Compass to view any data you and your applications have stored in your deployment. You can also use MongoDB Compass to create new databases, collections, and documents. Specific information can be found in the [MongoDB Compass documentation](https://docs.mongodb.com/compass/current/).





 Next Steps 



* For guidance on best practices, check out [Best Practices for MongoDB on the IBM Cloud](https://www.ibm.com/cloud/blog/best-practices-for-mongodb-on-the-ibm-cloud). If you are using MongoDB for the first time, see the [official MongoDB documentation](https://docs.mongodb.com/).
* Connect to and manage your MongoDB database through the [MongoDB Shell](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-connecting-cli-client) and explore the [OpsManager](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-ops-manager) functions offered in Databases for MongoDB Enterprise Edition deployments.
* Looking for more tools on managing your databases and data?
fs.writeFileSync('debug_modified.sh', scriptContentModified)
}
let finalBase64Content=''
if( needCompression)
{
const compressed = zlib.gzipSync(Buffer.from(scriptContentModified));
finalBase64Content = Buffer.from(compressed).toString('base64');
}
else
{
finalBase64Content =Buffer.from(scriptContentModified).toString('base64');
}

if(debugTempFile)
{
console.log('saving base64 encoded content as debug_modified_base64');
fs.writeFileSync('debug_modified_base64', finalBase64Content)
}

inputIGNJSON.storage.files[0].contents.source=${BASE64HEAD}${finalBase64Content}

console.log(saving final ign file to ${outputIGNFilePath});
fs.writeFileSync(outputIGNFilePath, JSON.stringify(inputIGNJSON));

Show more
5. Run the script.

node ign_tool.js attachHost-LOCATION.ign '{"tunnel_host":"INGRESS-SUBDOMAIN","location_id":"cdcnlr820p5snpe0bte0","auth_code":"70a65b5d6e94da10xxxx"}



Now you have the script. Depending on your infrastructure, you can create your Red Hat CoreOS Location.
* Always On availability groups provide the ability to keep a discrete set of databases highly available across one or more cluster nodes and work at the database level. Availability groups consist of one primary replica and up to a maximum of eight secondary replicas, and use synchronous or asynchronous data replication. In this deployment synchronous replication is used due to the low latency between the AZs.
* Distributed Network Names is a name resource in WSFC and Always On availability groups, used for name resolution of the cluster resources.



This deployment pattern does not leverage a Failover Cluster Instance (FCI). A key requirement of an FCI is that cluster nodes have access to shared storage. Shared storage is not native to IBM Cloud VPC environment and while its is possible to create shared storage, by using technologies such as Storage Spaces Direct. For further information Storage Spaces Direct, refer to [Storage Spaces Direct overview](https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview).



 Windows Server Failover Cluster 

Deploying Always On availability groups for HA on Windows requires a Windows Server Failover Cluster (WSFC). Each availability replica of an availability group must reside on a different node of the WSFC. To simplify security configuration for the availability databases, it is recommended that the SQL server instances (the service accounts) run under a domain service account.

Windows Server Failover Cluster (WSFC) is a Windows Server feature and each server that acts as a cluster node must have this feature enabled. WSFC relies on quorum votes to prevent "split-brain" syndrome, and there are a number of quorum modes:



* Node Majority – Active cluster nodes determine the quorum. At least half of the possible votes must be affirmative for the cluster to maintain a healthy state.
* Node and File Share Majority – A remote file share acts as a voting witness for the active nodes involved in a quorum vote. As with Node Majority, at least half of the possible votes must be affirmative for the cluster to maintain a healthy state.
* Node and Disk Majority – A shared disk acts as a voting witness, along with the active nodes involved in a quorum vote.
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 09:03:19 2023
* Last change: Tue Feb 14 09:01:40 2023 by s01adm via crm_resource on cl-sap-1
* 2 nodes configured
* 11 resource instances configured

Node List:
* Online: [ cl-sap-1 cl-sap-2 ]

Full List of Resources:
* res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-sap-1
* Resource Group: s01_ascs01_group:
* s01_vip_ascs01	(ocf::heartbeat:IPaddr2):	 Started cl-sap-2
* s01_fs_ascs01_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-2
* s01_fs_ascs01	(ocf::heartbeat:Filesystem):	 Started cl-sap-2
* s01_ascs01	(ocf::heartbeat:SAPInstance):	 Started cl-sap-2
* Resource Group: s01_ers02_group:
* s01_vip_ers02	(ocf::heartbeat:IPaddr2):	 Started cl-sap-1
* s01_fs_ers02_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-1
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-1
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-1
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 08:50:53 2023
* Last change: Tue Feb 14 08:50:50 2023 by hacluster via crmd on cl-sap-2
* 2 nodes configured
* 11 resource instances configured

Node List:
* Online: [ cl-sap-1 cl-sap-2 ]

Full List of Resources:
* res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-sap-1
* Resource Group: s01_ascs01_group:
* s01_vip_ascs01	(ocf::heartbeat:IPaddr2):	 Started cl-sap-1
* s01_fs_ascs01_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-1
* s01_fs_ascs01	(ocf::heartbeat:Filesystem):	 Started cl-sap-1
* s01_ascs01	(ocf::heartbeat:SAPInstance):	 Started cl-sap-1
* Resource Group: s01_ers02_group:
* s01_vip_ers02	(ocf::heartbeat:IPaddr2):	 Started cl-sap-2
* s01_fs_ers02_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-2
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-2
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-2
* s01_fs_ers02_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-1
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-1
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-1
* Clone Set: fs_sapmnt-clone [fs_sapmnt]:
* Started: [ cl-sap-1 ]
* Stopped: [ cl-sap-2 ]

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled





 Test2 - Recovery procedure 

Wait until NODE2 restarts, then restart the cluster framework.

On NODE1, run the following command.

pcs cluster start



* The cluster starts on NODE2 and acquires the ERS resources (virtual IP address, file system /usr/sap/${SID}/ERS${ERS_nr}, and the LVM resources) on NODE2.
* The cluster starts the ERS instance on NODE2.



Wait a moment and check the status with the following command. The ERS resource group moved to the second node.

pcs status

Sample output:

 pcs status
Cluster name: SAP_ASCS
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 08:41:23 2023
* Last change: Tue Feb 14 08:34:04 2023 by hacluster via crmd on cl-sap-1
* 2 nodes configured
* 11 resource instances configured

Node List:
* Online: [ cl-sap-1 cl-sap-2 ]

Full List of Resources:
* res_fence_ibm_powervs
When the production system assumes the PRIMARY role on NODE2 if a takeover occurs, the nonproduction system stops and its memory resources are released. The following cluster constraints make sure that the primary production instance and the nonproduction instance never run together on one node, and that the nonproduction instance stops before the production instance is promoted.

pcs constraint colocation add group_${sid_np}_non_prod with master SAPHana_${SID}_${INSTNO}-clone score=-INFINITY

pcs constraint order stop group_${sid_np}_non_prod then promote SAPHana_${SID}_${INSTNO}-clone

The cluster configuration is complete.

Run the following command to check the status of the defined cluster resources.

pcs status --full

Sample output:

 pcs status --full
Cluster name: SAP_PRD
Cluster Summary:
* Stack: corosync
* Current DC: cl-prd-2 (2) (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Fri Apr 28 16:38:00 2023
* Last change: Fri Apr 28 16:37:49 2023 by hacluster via crmd on cl-prd-1
* 2 nodes configured
* 8 resource instances configured

Node List:
* Online: [ cl-prd-1 (1) cl-prd-2 (2) ]

Full List of Resources:
* res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-prd-2
* Clone Set: SAPHanaTopology_PRD_00-clone [SAPHanaTopology_PRD_00]:
* SAPHanaTopology_PRD_00	(ocf::heartbeat:SAPHanaTopology):	 Started cl-prd-2
* SAPHanaTopology_PRD_00	(ocf::heartbeat:SAPHanaTopology):	 Started cl-prd-1
One of the last things shown in the log file after the Apply plan is an SSH key string:


2023/03/29 12:20:02 Terraform refresh | secondary_host_domain_name = "hpc-smc-secondary.smc.ibmhpc.com"
2023/03/29 12:20:02 Terraform refresh | smc_web_console = "https://localhost:8443/platform"
2023/03/29 12:20:02 Terraform refresh | ssh_command = "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -L 8443:localhost:8443 -J ubuntu@169.63.102.28 root@10.10.0.5"
2023/03/29 12:20:02 Command finished successfully.
OK


You use this string to configure SSH in the next step.





 Step 5: Configure ssh 

Copy the SSH command output from the Apply plan log to your laptop terminal. Use this ssh command to SSH to the primary node through the jump host public IP, and then to SSH into one of the nodes.

Use the public IP address of the jump host and modify the IP address of the target node to enable access to specific hosts through the jump host.





 Next steps 

After you provision the SMC cluster, wait for 10 min for the host data to be copied (rsync) to the secondary system. The configuration uses the primary SMC host.

Continue with making a connection between Lone Symphony and SMC Cluster
const { FileSystemWallet, Gateway } = require('fabric-network');
const fs = require('fs');
const path = require('path');

async function main() {
try {

// Parse the connection profile. This would be the path to the file downloaded
// from the IBM Blockchain Platform operational console.
const ccpPath = path.resolve(__dirname, 'connection.json');
const ccp = JSON.parse(fs.readFileSync(ccpPath, 'utf8'));

// Configure a wallet. This wallet must already be primed with an identity that
// the application can use to interact with the peer node.
const walletPath = path.resolve(__dirname, 'wallet');
const wallet = new FileSystemWallet(walletPath);

// Create a new gateway, and connect to the gateway peer node(s). The identity
// specified must already exist in the specified wallet.
const gateway = new Gateway();
await gateway.connect(ccp, { wallet, identity: 'user1' , discovery: {enabled: true, asLocalhost:false }});

// Get the network channel that the smart contract is deployed to.
const network = await gateway.getNetwork('<channel_name>');

// Get the smart contract from the network channel.
const contract = network.getContract('<smart_contract_name>');

// Submit the 'createCar' transaction to the smart contract, and wait for it
// to be committed to the ledger.
await contract.submitTransaction('createCar', 'CAR12', 'Honda', 'Accord', 'Black', 'Tom');
console.log('Transaction has been submitted');

await gateway.disconnect();

} catch (error) {
* Resources ${sid}_vip_ascs${ASCS_nr}, ${sid}_fs_ascs${ASCS_nr}_lvm, ${sid}_fs_ascs${ASCS_nr} and ${sid}_ascs${ASCS_nr} are Started on NODE1.
* Resource group ${sid}_ers${ERS_nr}_group is active on NODE2.
* Resources ${sid}_vip_ers${ERS_nr}, ${sid}_fs_ers${ERS_nr}_lvm, ${sid}_fs_ers${ERS_nr} and ${sid}_ers${ERS_nr} are Started on NODE2.



* Check SAP instance processes:



* ASCS instance is running on NODE1.
* ERS instance is running on NODE2.





pcs status

Sample output:

 pcs status
Cluster name: SAP_ASCS
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 07:59:16 2023
* Last change: Tue Feb 14 05:02:22 2023 by hacluster via crmd on cl-sap-1
* 2 nodes configured
* 11 resource instances configured

Node List:
* Online: [ cl-sap-1 cl-sap-2 ]

Full List of Resources:
* res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-sap-2
* Resource Group: s01_ascs01_group:
* s01_vip_ascs01	(ocf::heartbeat:IPaddr2):	 Started cl-sap-1
* s01_fs_ascs01_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-1
* s01_fs_ers02_lvm	(ocf::heartbeat:LVM-activate):	 Started cl-sap-2
* s01_fs_ers02	(ocf::heartbeat:Filesystem):	 Started cl-sap-2
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-2
* Clone Set: fs_sapmnt-clone [fs_sapmnt]:
* Started: [ cl-sap-1 cl-sap-2 ]

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled





 Test3 - Recovery Procedure 

On NODE2, run the following commands.

pcs resource refresh

pcs status --full







 Test4 - Testing the manual move of the ASCS instance 

Use the folling information to test a manual move of an ASCS instance.



 Test4 - Description 

Use SAP Control commands to move the ASCS instance to the other node for maintenance purposes.





 Test4 - Prerequisites 



* A functional two-node RHEL HA Add-On cluster for SAP ENSA2.
* The sap_cluster_connector is installed and configured.
* Both cluster nodes are active.
* Cluster is started on NODE1 and NODE2.



* Resource group ${sid}_ascs${ASCS_nr}_group is active on NODE1.
* Resources ${sid}_vip_ascs${ASCS_nr}, ${sid}_fs_ascs${ASCS_nr}_lvm, ${sid}_fs_ascs${ASCS_nr} and ${sid}_ascs${ASCS_nr} are Started on NODE1.
* Resource group ${sid}_ers${ERS_nr}_group is active on NODE2.
As with Node Majority, at least half of the possible votes must be affirmative for the cluster to maintain a healthy state.
* Node and Disk Majority – A shared disk acts as a voting witness, along with the active nodes involved in a quorum vote.
* Disk Only – A shared disk acts as a witness and quorum is determined by which nodes can access the disk. There is no minimum number of possible votes required.



The Microsoft recommendation for SQL server are as follows:



* Use Node Majority quorum mode when there is an odd number of voting nodes.
* Use the Node and File Share Majority quorum mode when you have an even number of voting nodes.







 Always On Availability Groups 

An availability group supports one set of primary databases and up to eight sets of secondary databases. Secondary databases are not backups, so it is essential that you back up the databases and their transaction logs on a regular basis. Always On availability groups require one of the following three cluster type options; WSFC, EXTERNAL, NONE.



* WSFC - Uses Windows Server Failover Cluster (WSFC) to manage the cluster failover. This is a prerequisite for high availability groups on Windows servers.
* External - Uses an external cluster manager. For example, SQL Server on Linux supports Pacemaker. Availability groups on Linux clusters require at least two synchronous replicas to guarantee HA, but at least three replicas for automatic recovery, so it is recommend that an availability group is setup on at least three nodes.
* None - Known as clusterless availability groups or was initially referred to as a read-scale availability group. However, this option provides only a subset of availability groups features, and does not include automatic failover. The features do include manual planned and forced failover and both synchronous and asynchronous replication modes, readable secondary nodes, and secondary replica backups. Availability groups without a WSFC cluster can still provide readable secondary nodes, read-only routing, and load balancing. Failover can be automated with external tools. A new feature of SQL Server 2019 helps mitigate this with automatic traffic redirection, which provides for routing of read/write traffic back to the primary node and does not require a Listener.
Started cl-sap-2
* s01_ers02	(ocf::heartbeat:SAPInstance):	 Started cl-sap-2
* Clone Set: fs_sapmnt-clone [fs_sapmnt]:
* Started: [ cl-sap-1 cl-sap-2 ]

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled







 Test3 - Testing failure of the ERS instance 

Use the following information to test the failure of an ERS instance.



 Test3 - Description 

Simulate a crash of the ERS instance.





 Test3 - Prerequisites 



* A functional two-node RHEL HA Add-On cluster for SAP ENSA2.
* Both cluster nodes are active.
* Cluster starts on NODE1 and NODE2.



* Resource group ${sid}_ascs${ASCS_nr}_group is active on NODE1.
* Resources ${sid}_vip_ascs${ASCS_nr}, ${sid}_fs_ascs${ASCS_nr}_lvm, ${sid}_fs_ascs${ASCS_nr} and ${sid}_ascs${ASCS_nr} are Started on NODE1.
* Resource group ${sid}_ers${ERS_nr}_group is active on NODE2.
* Resources ${sid}_vip_ers${ERS_nr}, ${sid}_fs_ers${ERS_nr}_lvm, ${sid}_fs_ers${ERS_nr} and ${sid}_ers${ERS_nr} are Started on NODE2.



* Check SAP instance processes:



* ASCS instance is running on NODE1.
* ERS instance is running on NODE2.









 Test3 - Test Procedure 

Crash SAP ERS instance by sending a SIGKILL signal.

On NODE2, identify the PID of the enque replication server.
* Resources ${sid}_vip_ers${ERS_nr}, ${sid}_fs_ers${ERS_nr}_lvm, ${sid}_fs_ers${ERS_nr} and ${sid}_ers${ERS_nr} are Started on NODE2.



* Check SAP instance processes:



* ASCS instance is running on NODE1.
* ERS instance is running on NODE2.









 Test3 - Test Procedure 

Crash SAP ERS instance by sending a SIGKILL signal.

On NODE2, identify the PID of the enque replication server.

pgrep -af "(er|enqr).sap"

Send a SIGKILL signal to the identified process.

Sample output:

 pgrep -af "(er|enqr).sap"
2527198 er.sapS01_ERS02 pf=/usr/sap/S01/ERS02/profile/S01_ERS02_cl-sap-ers NR=01
 kill -9 2527198





 Test3 - Expected behavior 



* SAP Enqueue Replication Server on NODE2 crashes immediately.
* The cluster detects the stopped ERS and marks the resource as failed.
* The cluster restarts the ERS on NODE2.



Check the status with the following command.

pcs status

The ${sid}_ers${ERS_nr}ERS resource restarted on the second node. If you run the pcs status command too soon, you might see the ERS resource briefly in status FAILED.

Sample output:

 pcs status
Cluster name: SAP_ASCS
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 08:50:53 2023
* Last change: Tue Feb 14 08:50:50 2023 by hacluster via crmd on cl-sap-2
* 2 nodes configured
The destination folders should be empty. If they're not, empty them.

rsync -av /mnt/local-storage/storage/watson/assistant/{yourPV} {other worker}:/mnt/local-storage/storage/watson/assistant
* Double-check the postgres-keeper persistent volume permissions on each worker node.

You might need to scale postgres-keeper deployments first to see which persistent volumes they claim. Connect over SSH to a worker node, and then run the following command:

kubectl get pvc

For each postgres-keeper persistent volume claim, change the permissions of /postgres to u=rwx (0700) on its given worker node.

chmod 0700 /mnt/local-storage/storage/watson/assistant/{yourPV}/postgres



2. If you need to stop and restart the underlying OpenShift cluster for any reason, you can do so now.

For more information, see [How To: Stop and start a production OpenShift Cluster](https://servicesblog.redhat.com/2019/05/29/how-to-stop-and-start-a-production-openshift-cluster/).
3. Scale the service back up by scaling up the deployed services in the following order:



* All statefulsets (except postgres-store-keeper)
* postgres-store-sentinel deployment
* postgres-store-proxy deployment
* postgres-store-keeper statefulset
* All deployments (except gateway, ui, and store)
* gateway deployment
* store deployment
* ui deployment











 To view logs 

Consider using [Cognos Dashboards](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=services-cognos-dashboards) to identify patterns in your data.





 Managing user access 

After you provision an instance, you can share the URL for the product user interface with other users.
* Resources ${sid}_vip_ers${ERS_nr}, ${sid}_fs_ers${ERS_nr}_lvm, ${sid}_fs_ers${ERS_nr} and ${sid}_ers${ERS_nr} are Started on NODE1.



* Check SAP instance processes:



* ASCS instance is running on NODE2.
* ERS instance is running on NODE1.









 Test2 - Test procedure 

Crash primary on NODE2 by sending a fast-restart system request.

On NODE2, run the following command.

sync; echo b > /proc/sysrq-trigger





 Test2 - Expected behavior 



* NODE2 restarts.
* The cluster detects the failed node and sets its state to offline (UNCLEAN).
* The cluster acquires the ASCS resources (virtual IP address, file system /usr/sap/${SID}/ASCS${ASCS_nr}, and the LVM items) on NODE1.
* The cluster starts the ASCS on NODE1.
* The cluster stops the ERS instance on NODE1.
* The cluster stops the dependent resources on NODE1 (virtual IP address, file system /usr/sap/${SID}/ERS${ERS_nr}, and the LVM resources), and releases them.



After a while, check the status with the following command.

The second node is offline and both resource groups are running on the first node.

pcs status

Sample output:

 pcs status
Cluster name: SAP_ASCS
Cluster Summary:
* Stack: corosync
* Current DC: cl-sap-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
* Last updated: Tue Feb 14 08:34:16 2023
* Last change: Tue Feb 14 08:34:04 2023 by hacluster via crmd on cl-sap-1
* 2 nodes configured
* 11 resource instances configured

Node List: