vania-janet/multiturn-rag-retrieval-data huggingface dataset:
oc get pods -o wide -n openshift-storage| egrep -i <new_node_name> | egrep osd
4. Identify the crashcollector pod deployment.

oc get deployment --selector=app=rook-ceph-crashcollector,node_name=NODE-NAME -n openshift-storage
5. If there is an existing crashcollector deployment, delete it.

oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=NODE-NAME -n openshift-storage
6. Delete the ocs-osd-removal-job.

oc delete -n openshift-storage job ocs-osd-removal-job

Example output:

job.batch "ocs-osd-removal-job" deleted







 Step 8: Update the OpenShift Data Foundation add-on 

Major



1. Check the existing version.

ibmcloud oc cluster addon ls --cluster CLUSTER
2. Update the add-on.

ibmcloud oc cluster addon update --addon openshift-data-foundation --cluster CLUSTER --version VERSION
3. Verify the add-on is updated.

ibmcloud oc cluster addon ls --cluster CLUSTER







 Step 9: Update your cluster resource 

Major



1. Get the name of your ocscluster resource.

oc get ocscluster

Example output

NAME AGE
ocscluster-vpc 19d
2. Run the following command to edit your ocscluster resource.

oc edit ocscluster OCS-CLUSTER-NAME
3. Set the ocsUpgrade parameter to true.

...
spec:
billingType: hourly
monSize: 20Gi
autoDiscoverDevices: true
numOfOsd: 1
ocsUpgrade: true
osdSize: 100Gi
status:
storageClusterStatus: Decreasing the capacity not allowed
In the following example output from the lsblk command, the sdc disk has two available, unformatted partitions that you can use for the OSD and MON device paths for this worker node. If your worker node has raw disks without partitions, you need one disk for the OSD and one disk for the MON. As a best practice, and to maximize storage capacity on this disk, you can specify the smaller partition or disk for the MON, and the larger partition or disk for the OSD. Note that the initial storage capacity of your ODF configuration is equal to the size of the disk that you specify as the osd-device-path when you create your configuration.

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931G 0 disk
|-sda1 8:1 0 256M 0 part /boot
|-sda2 8:2 0 1G 0 part
-sda3 8:3 0 929.8G 0 part /
sdb 8:16 0 744.7G 0 disk
-sdb1 8:17 0 744.7G 0 part /disk1
sdc 8:32 0 744.7G 0 disk
|-sdc1 8:33 0 18.6G 0 part
-sdc2 8:34 0 260.8G 0 part
5. Find the by-id for each disk that you want to use in your configuration. In this case, the sdc1 and sdc2 partitions are unformatted and unmounted. The by-id for each disk is specified as a command parameter when you create your configuration.

ls -l /dev/disk/by-id/

If you have VMware hosts, run the following command.

ls -l /dev/disk/by-path/
6. Review the command output and make a note of the by-id values for the disks that you want to use in your configuration.
Organizing hardware nodes in Virtuozzo 



1.  Go to Infrastructure > Hardware Nodes.
2.  Clock Add New Subfolder.
3.  Give the folder an appropriate title and select either Folder, Datacenter, or Rack. In this example you create three datacenter folders within a folder.
4.  Return to Infrastructure > Hardware Nodes.
5.  Select the nodes that you want to move and click Move.
6.  Select the destination and click Submit.
oc scale deployment rook-ceph-osd-2 --replicas=0 -n openshift-storage
deployment.apps/rook-ceph-osd-2 scaled
oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=NODE-NAME --replicas=0 -n openshift-storage







 Step 4: Cordon and drain the worker node 

Major

Minor



1. Cordon the node. Cordoning the node prevents any pods from being scheduled on this node.

oc adm cordon NODE_NAME

Example output

node/10.241.0.4 cordoned
2. Drain the node to remove all the pods. When you drain the worker node, the pods move to the other worker nodes ensuring there is no downtime. Draining also ensures that there is no disruption of the pod disruption budget.

oc adm drain NODE_NAME --force --delete-emptydir-data --ignore-daemonsets

Example output

evicting pod "managed-storage-validation-webhooks-7fd79bc9f7-pdpv6"
evicting pod "calico-kube-controllers-647dbbd685-fmrp9"
evicting pod "certified-operators-2v852"
evicting pod "csi-snapshot-controller-77fbf474df-47ddt"
evicting pod "calico-typha-8574d89b8c-7f2cc"
evicting pod "dns-operator-6d48cbff67-vrrsw"
evicting pod "router-default-6fc798b98b-9m6kh"
evicting pod "prometheus-adapter-5b77ffdd5f-hzqrp"
evicting pod "alertmanager-main-1"
evicting pod "prometheus-k8s-0"
* From the main IBM Cloud Pak for Data web client navigation menu, select My instances.
* On the Provisioned instances tab, click your Watson Assistant instance.
* In the Access information of the instance, find the Bearer token. Copy the token and paste it into the accessTokens list.



A bearer token for an instance can access all instances that are owned by the user. Therefore, if a single user owns all of the instances, then only one bearer token is required.

If the service has multiple instances, each owned by a different user, then you must gather bearer tokens for each user who owns an instance. You can list multiple bearer token values in the accessTokens section.
2. To get the host information, you need details for the pod that hosts the Watson Assistant UI component:

oc describe pod -l component=ui

Look for the section that says, RESOURCE_CONTROLLER_URL: https://${release-name}-addon-assistant-gateway-svc.zen:5000/api/ibmcloud/resource-controller

For example, you can use a command like this to find it:

oc describe pod -l component=ui | grep RESOURCE_CONTROLLER_URL

Copy the host that is specified in the RESOURCE_CONTROLLER_URL. The host value is the RESOURCE_CONTROLLER_URL value, excluding the protocol at the beginning and everything from the port to the end of the value. For example, for the previous example, the host is ${release-name}-addon-assistant-gateway-svc.zen.
3. To get the port information, again check the RESOURCE_CONTROLLER_URL entry. The port is specified after <host>: in the URL. In this sample URL, the port is 5000.
4. Paste the values that you discovered into the YAML file and save it.







 Creating the postgres.yaml file 

The postgres.yaml file contains details about the Postgres pods in your target environment (the environment where you will restore the data). Add the following information to the file:

host: localhost
port: 5432
database: store
username: user
get-synonym Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, value, synonym, include_audit Get an entity value synonym. 
 list-synonyms Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, value, page_limit, include_count, sort, cursor, include_audit List entity value synonyms. 
 update-synonym Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, value, synonym, new_synonym Update an entity value synonym. 
 create-dialog-node Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, dialog_node, description, conditions, parent, previous_sibling, output, context, metadata, next_step, actions, title, node_type, event_name, variable, digress_in, digress_out, digress_out_slots Create a dialog node. 
 delete-dialog-node Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, dialog_node Delete a dialog node. 
 get-dialog-node Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, dialog_node, include_audit Get a dialog node.
* df -H # Dump disk usage information
* vmstat # Dump memory usage information
* lshw # Dump hardware information
* last -Fxn2 shutdown reboot # determine if last restart was graceful or not
* mount | grep -i "(ro" # to rule out disk read-only issue. NOTE: tmpfs being ro is fine
* touch /this # to rule out disk read-only issue





5. [Open a support ticket](https://cloud.ibm.com/unifiedsupport/cases/form) and attach all the outputs saved in the previous steps.
To find the device path after attaching the disk, log in to your worker node with oc debug <node> and run lsblk to list the devices on that node.



4. Continue with your Portworx setup by [Setting up a key-value store for Portworx metadata](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_portworx_kv_store).







 Satellite clusters 

Satellite



1. [Create a Satellite location](https://cloud.ibm.com/docs/satellite?topic=satellite-locations).
2. Create VMs in your cloud provider or set up on-premises hosts that have at least 16 vCPU and 64 GB and enough storage to be attached to your location plus additional storage volumes for app data, the Portworx journal, and KVDB. For highly available data storage, Portworx requires at least 3 worker nodes with raw and unformatted block storage.
3. If you want to use [journal devices](https://docs.portworx.com/install-with-other/operate-and-maintain/performance-and-tuning/tuning/), choose from the following options. Journal devices allow Portworx to write logs directly to a local disk on your worker node.



* Attach an additional 3 GB disk to at least 3 worker nodes in your cluster and find the device path. To find the device path after you attach the disk, log in to your worker node with oc debug <node> and run lsblk to list the devices on that node.
* Select a device on a worker node where you want Portworx to create the journal.



4. If you want to use a specific device for the internal Portworx KVDB, choose from the following options.



* Attach an additional disk to at least 3 worker nodes use for the KVDB, add a 3 GB block storage device to at least 3 worker nodes in your cluster and find the device path.
* [Node selector](https://docs.openshift.com/container-platform/4.11/logging/config/cluster-logging-moving-nodes.html). Add the node selector to the Elasticsearch (logstore)and Kibana (visualization), and Fluentd (collector.logs) pods.

spec:
logStore:
elasticsearch:
nodeSelector:
logging: clo-efk
...
visualization:
kibana:
nodeSelector:
logging: clo-efk
...
collection:
logs:
fluentd:
nodeSelector:
logging: clo-efk
Show more
* [Toleration](https://docs.openshift.com/container-platform/4.11/logging/config/cluster-logging-tolerations.html). Add the node selector to the Elasticsearch (logstore)and Kibana (visualization), and Fluentd (collector.logs) pods.

spec:
logStore:
elasticsearch:
tolerations:
- key: app
value: clo-efk
operator: "Exists"
effect: "NoExecute"
...
visualization:
kibana:
tolerations:
- key: app
value: clo-efk
operator: "Exists"
effect: "NoExecute"
...
collection:
logs:
fluentd:
tolerations:
- key: app
value: clo-efk
operator: "Exists"
effect: "NoExecute"
Show more



8. Click Create.
9. Verify that the operator, Elasticsearch, Fluentd, and Kibana pods are all Running.
"interpretation": {
"value": string,
"numeric_value": number,
"unit": string,
},
"provenance_ids": string, string, ... ],
"location": { "begin": int, "end": int }
},
...
],
"payment_terms": [
{
"confidence_level": string,
"text": string,
"text_normalized": string,
"interpretation": {
"value": string,
"numeric_value": number,
"unit": string,
},
"provenance_ids": string, string, ... ],
"location": { "begin": int, "end": int }
},
...
],
"contract_currencies": [
{
"confidence_level" : string,
"text" : string,
"text_normalized" : string,
"provenance_ids": string, string ..],
"location": { "begin": int, "end": int }
},
...
],
"tables": [
{
"location" : {
"begin" : int,
"end" : int
},
"text": string,
"section_title": {
"text": string,
"location": {
"begin" : int,
"end" : int
}
},
"title": {
"location": {
"begin": int,
"end": int,
},
"text": string
},
"table_headers" :
{
"cell_id" : string,
"location" : {
"begin" : int,
"end" : int
},
"text" : string,
On NODE1, run the following command.

pcs status

Then, log in to the identified cluster node and send a shutoff system request.

sync; echo o > /proc/sysrq-trigger





 Test2 - Expected behavior 



* The node with the active nfsgroup resource group shuts down.
* The cluster detects the failed node and starts a fencing action.
* The fencing operation sets the state of the fenced node to offline.
* The cluster acquires the NFS Server resources on the failover node.



Check that all the resources started on the failover node.

pcs resource status

Sample output:

 pcs resource status
* Resource Group: nfsgroup:
* nfs_vg (ocf::heartbeat:LVM-activate): Started cl-nfs-2
* nfs_fs1 (ocf::heartbeat:Filesystem): Started cl-nfs-2
* nfs_fs2 (ocf::heartbeat:Filesystem): Started cl-nfs-2
* nfs_fs3 (ocf::heartbeat:Filesystem): Started cl-nfs-2
* nfs_daemon (ocf::heartbeat:nfsserver): Started cl-nfs-2
* nfs_export (ocf::heartbeat:exportfs): Started cl-nfs-2
* nfs_ip (ocf::heartbeat:IPaddr2): Started cl-nfs-2
* nfs_notify (ocf::heartbeat:nfsnotify): Started cl-nfs-2

Verify that the file system is still mounted on the NFS client node, and that the applications can still access the content.





 Test2 - Recovery procedure 

Log in to the IBM Cloud® Console and start the stopped instance. Wait until the cluster node is available again, then restart the cluster framework.

On the cluster node, run the following command.

pcs cluster start

Check the cluster status.

pcs status

Sample output:

 pcs status
Cluster name: SAP_NFS
Cluster Summary:
15","u26"],"u34","u40","d16","u31","u35","m16"],"u38","m17","d17","u36"],"u37","u33","u44","u38","d18","u39","m18","u36"],"u42","u40","d19","u35","u43","u46","u41","m19"],"u43","u41","m20","d20"],"u44","d21","m21","u48","u39","u45"],"m22","u42","u47","u52","u46","d22"],"u50","d23","u49","u48","m23","u45"],"u50","m24","d24","u53","u49","u51"],"d25","u54","u47","m25","u53","u52","u51","u55"],"d26","u54","m26","
The certificate can be confirmed again by removing the HPVS instance from file /Users/hexiaohu/.hpnet-cfg/known_instances

Continue connecting to the server...

Refreshing the node hpnet-test1 ...

The refreshed connections:
+---------------+-------------+-----------+--------------+-------+
 INSTANCE_NAME TARGET_NAME STATE SYNCHRONIZED ERROR 
+---------------+-------------+-----------+--------------+-------+
 hpnet-test1 hpnet-test2 connected true 
+---------------+-------------+-----------+--------------+-------+

OK
Node hpnet-test1 was refreshed.
Show more
* The hpnet node delete command, to clear the configuration of the VPN node

ibmcloud hpnet node delete --instance "$HPVS_A" --admin-priv-key "$ADMIN_PRIV_KEY_A"

Enter passphrase to use the private key '--admin-priv-key'>
OK

Starting to decrypt key '/Users/hexiaohu/.hpnet-cfg/hpnet-test1-c3-10-f7-a3.priv'....

Connecting to the server hpnet-test1 [52.117.54.147] ...

The connecting server's certificate is valid and already confirmed in the previous command
To learn how to configure the IPMI exporter, see [Configuring the Prometheus IPMI Exporter to monitor sensor metrics](https://cloud.ibm.com/docs/monitoring?topic=monitoring-ipmi).

You can also check the tutorial: [Configure the Prometheus IPMI Exporter to monitor sensor metrics in a Bare metal](https://cloud.ibm.com/docs/monitoring?topic=monitoring-baremetal_linuxbaremetal_linux_step3).





 Blackbox exporter 

Allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. The monitoring agent can be used in conjunction with the Blackbox exporter to collect availability metrics.

Configure the [Prometheus Blackbox exporter](https://github.com/prometheus/blackbox_exporter) to monitor host availability, such as URL sites.





 Node exporter 

In addition to the set of metrics that are automatically collected by the monitoring agent, you might want to collect other Linux host metrics that are exposed by *NIX kernels.

Configure the [Prometheus Node exporter](https://github.com/prometheus/node_exporter) to collect hardware and kernel metrics that are exposed by *NIX kernels.

For more information, see [Monitoring Linux host metrics with the Node Exporter](https://prometheus.io/docs/guides/node-exporter/).
If you didn't install NodeJS in your Jenkins environment, you might have to install the node.js Jenkins plug-in. If you installed NodeJS, you can comment out the "tools" section in the Jenkins file.





 Viewing Build Frequency 

When this build job completes, the pipeline publishes a message to DevOps Insights that a build is complete. You can view the build record on the Build Frequency page. To view the Build Frequency page, use the following steps.



1. From the IBM Cloud console, click the menu icon ![hamburger icon](https://cloud.ibm.com/docs-content/v1/content/562806c0b18119f4d21cb66f5ae6d051634b4a16/ContinuousDelivery/images/icon_hamburger.svg), and select DevOps.
2. Select your toolchain.
3. From your toolchain's Overview page, on the IBM Cloud tools card, click DevOps Insights.
4. Click Build Frequency.







 Next steps 

Learn how to [publish a deployment record](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-publish-deploy-jenkins) by using Jenkins.
Synchronizing High Availability (HA) configurations 

Two IBM Cloud® Virtual Router Appliances (VRAs) in a High Availability (HA) pair must have their configurations synchronized sufficiently so that both devices behave in a similar manner. This is done through configuration sync-maps and you can choose which portion of the configuration to synchronize. If you make a change on one machine, it pushes the marked config over to the other device.

This synchronizes and saves the running configuration of the local device on the remote device. However, as a step of the commit process, it does not save the configuration on the local device.

Configurations that are unique to one system should not be synced to the other. For instance, real IP addresses and MACs should not be synchronized. The system config-sync configuration node itself and the service https node cannot be synchronized.

The following example illustrates config-sync:

set system config-sync sync-map TEST rule 2 action include
set system config-sync sync-map TEST rule 2 location security firewall
set system config-sync remote-router 192.168.1.22 username vyatta
set system config-sync remote-router 192.168.1.22 password xxxxxx
set system config-sync remote-router 192.168.1.22 sync-map TEST

The first two lines create the actual sync-map itself. Here, the configuration stanza for security firewall is set in the sync-map. As a result, any changes made inside the config node are pushed to the remote device. However, changes made to security user would not be sent, as that doesn't match the rule. You can make the sync-map as specific, or as general as you want.

The next three lines designate the remote router's config-sync user and password, IP, and which sync-map to push. Any changes that match the rules for TEST, go to remote-router 192.168.1.22, using this login information. Note that for version 1801zf and earlier, a REST call is made to perform this using the VRA API.
+---------------+-------------+----------+-----------+--------------+
 INSTANCE_NAME TARGET_NAME TYPE STATE SYNCHRONIZED 
+---------------+-------------+----------+-----------+--------------+
 hpnet-test1 hpnet-test2 outgoing connected true 
+---------------+-------------+----------+-----------+--------------+

OK
Show more
* The hpnet connect sync command, to synchronize the connection configuration which might be refreshed but out of synchronization on any of the VPN node:

ibmcloud hpnet connect sync --instance "$HPVS_A" --target "$HPVS_B" --admin-priv-key "$ADMIN_PRIV_KEY_A"

Enter passphrase to use the private key '--admin-priv-key'>
OK

Starting to decrypt key '/Users/hexiaohu/.hpnet-cfg/hpnet-test1-c3-10-f7-a3.priv'....

Connecting to the server hpnet-test1 [52.117.54.147] ...

The connecting server's certificate is valid and already confirmed in previous command
The certificate can be confirmed again by removing the HPVS instance from file /Users/hexiaohu/.hpnet-cfg/known_instances

Continue connecting to the server...
* Clone Set: SAPHanaTopology_PRD_00-clone [SAPHanaTopology_PRD_00]:
* SAPHanaTopology_PRD_00	(ocf::heartbeat:SAPHanaTopology):	 Started cl-prd-2
* SAPHanaTopology_PRD_00	(ocf::heartbeat:SAPHanaTopology):	 Started cl-prd-1
* Clone Set: SAPHana_PRD_00-clone [SAPHana_PRD_00] (promotable):
* SAPHana_PRD_00	(ocf::heartbeat:SAPHana):	 Slave cl-prd-2
* SAPHana_PRD_00	(ocf::heartbeat:SAPHana):	 Master cl-prd-1
* vip_PRD_00	(ocf::heartbeat:IPaddr2):	 Started cl-prd-1
* Resource Group: group_dev_non_prod:
* vip_np	(ocf::heartbeat:IPaddr2):	 Started cl-prd-2
* SAPHana_np_DEV_HDB10	(ocf::heartbeat:SAPInstance):	 Started cl-prd-2

Node Attributes:
* Node: cl-prd-1 (1):
* hana_prd_clone_state 	: PROMOTED
* hana_prd_op_mode 	: logreplay
* hana_prd_remoteHost 	: cl-prd-2
* hana_prd_roles 	: 4:P:master1:master:worker:master
* hana_prd_site 	: SiteA
* hana_prd_srmode 	: syncmem
* hana_prd_sync_state 	: PRIM
* hana_prd_version 	: 2.00.070.00.1679989823
* hana_prd_vhost 	: cl-prd-1
* Subscription identity sync
* Apply updates your worker nodes in place



* [3 October 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-oct322)



* CLI change log





* [September 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-sep22)



* [23 September 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-sep2322)



* Added support for remote workers



* [21 September 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-sep2122)



* CLI change log



* [12 September 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-sep1222)



* CLI change log



* [6 September 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-sep622)



* New! Location overview information.





* [August 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-aug22)



* [31 August 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-aug3122)



* New! You can now create Red Hat CoreOS enabled Satellite locations in the Sao Paulo (sao, br-sao) region.



* [23 August 2022](https://cloud.ibm.com/docs/satellite?topic=satellite-satellite-relnotessatellite-aug2322)



* New! You can now create Red Hat CoreOS enabled Satellite locations in the Osaka (osa, jp-osa) region.
In-sync replicas 

A follower replica that is keeping up with the partition leader is in-sync. Any follower with an in-sync replica can become the leader without losing any messages.

If the partition leader fails, another leader is chosen from the followers. All the replicas are to be usually in-sync. It is acceptable for a replica to be temporarily not in-sync while it is catching up after a failure.

Zoom

![In-sync-replicas diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/concepts_in_sync_replicas.png)

Figure 5. The diagram shows a topic partition that is replicated across three brokers and staying in-sync across all replicas.





 Producers 

A producer publishes messages to one or more topics. A producer can publish to one or more topics and can optionally choose the partition that stores the data.

You can also configure your producer to prioritize speed or reliability by choosing the level of acknowledgment the producer receives for messages it publishes.

For more information, see [Producing messages](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-producing_messagesproducing_messages).

Zoom

![Producers diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/concepts_producers.png)

Figure 6. The diagram shows a producer publishing messages to one topic across three brokers.





 Consumers 

A consumer reads messages from one or more topics and processes them. The difference between a consumer's current position and the newest message on a partition is known as the offset lag.

If the lag increases over time, it is a sign that the consumer is not able to keep up.