vania-janet/multiturn-rag-retrieval-data huggingface dataset:
It automates to securely forward private keys to remote nodes. Forwarded keys are deleted immediately after a session is closed.

To securely forward private keys to remote nodes, you need to do ssh-add and ssh -A.

[your local PC]$ ssh-add {id_rsa for lsf cluster}
[your local PC] ssh -A -J root@jumpbox_fip root@management_private_ip
...
[root@management] ssh -A worker_private_ip

For Mac OS X, you can persist ssh-add by adding the following configuration to .ssh/config:

Host 
  UseKeychain yes
  AddKeysToAgent yes

You can even remove -A by adding "ForwardAgent yes" to .ssh/config.
* How many worker nodes can I deploy in my Spectrum LSF cluster through this offering?

Before deploying a cluster, it is important to ensure that the VPC resource quota settings are appropriate for the size of the cluster that you would like to create (see [Quotas and service limits](https://cloud.ibm.com/docs/vpc?topic=vpc-quotas)).

The maximum number of worker nodes that are supported for the deployment value worker_node_max_count is 500 (see [Deployment values](https://cloud.ibm.com/docs/ibm-spectrum-lsf?topic=ibm-spectrum-lsf-deployment-values)). The worker_node_min_count variable specifies the number of worker nodes that are provisioned at the time the cluster is created, which will exist throughout the life of the cluster. The delta between those two variables specifies the maximum number of worker nodes that can either be created or destroyed by the LSF resource connector auto scaling feature. In configurations where that delta exceeds 250, it's recommended to take caution if the characteristics of the workload are expected to result in >250 cluster node join or remove operation requests at a single point in time. In those cases, it's recommended to pace the job start and stop requests, if possible. Otherwise, you might see noticeable delays in some subset of the nodes joining or being removed from the cluster.
If you require a higher level of service for automatic disaster recovery, see IBM's [Cloud disaster recovery solutions](https://www.ibm.com/cloud/disaster-recovery).
* What is restoring a volume from a snapshot?

 What is restoring a volume from a snapshot? 

Restoring from a snapshot creates a new, fully provisioned boot or data volume. You can restore storage volumes during instance creation, instance modification, or when you provision a new stand-alone volume. For data volumes, you can also use the volumes API to create a data volume from a snapshot. For more information, see [Restoring a volume from a snapshot](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-restore).

For best performance, you can enable snapshots for fast restore. By using the fast restore feature, you can create a volume from a snapshot that is fully provisioned when the volume is created. For more information, see [Snapshots fast restore](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-aboutsnapshots_vpc_fast_restore).
* Can I add tags to a volume?

 Can I add tags to a volume? 

Yes, you can add user tags and access management tags to your volumes. User tags are used by the backup service to automatically create backup snapshots of the volume. Access management tags help organize access to your Block Storage for VPC volumes. For more information, see [Tags for Block Storage for VPC volumes](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-about&interface=uistorage-about-tags).
* What is IOPS and how do it relate to my Block Storage for VPC volume performance?

 What is IOPS and how do it relate to my Block Storage for VPC volume performance? 

Input/output operations per second (IOPS) is used to measure the performance of your Block Storage for VPC volumes.
To prevent multiple clients from unknowingly overwriting each other's updates, select API methods support entity-tags and conditional requests.

This feature is now generally available. See the [VPC API change log](https://cloud.ibm.com/docs/vpc?topic=vpc-api-change-log22-march-2022).







 23 November 2021 



 For all version dates 

Customer-managed encryption for file shares. Accounts that have been granted special approval to preview this feature can now use customer-managed encryption, also called Bring Your Own Key (BYOK), to [create a file share](https://cloud.ibm.com/apidocs/vpc-betacreate-share) that is encrypted using your root key. Specify the new encryption_key property and crn sub-property of the root key that you either imported to IBM Cloud or created in Key Protect or Hyper Protect Crypto Services (HPCS). For more information, see [Creating file shares with customer-managed encryption](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-vpc-encryption&interface=api).

You can also use the API to rotate the root keys that are protecting your file shares. See [Key rotation for VPC resources](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-key-rotation&interface=apivpc-key-rotation-api-procedure) for details.

Supplemental user/group IDs for file shares. Accounts that have been granted special approval to preview this feature can now access supplemental user/group IDs for file shares. When a process runs on Unix/Linux, the operating system identifies a user with a user ID (UID) and/or group with a group ID (GID). These IDs determine which system resources a user or group can access. When you [create a file share](https://cloud.ibm.com/apidocs/vpc-betacreate-share), you can specify the new initial_owner property and specify a uid, gid, or both sub-properties to control access to the share.
As with anything that is deployed to a Kubernetes-based cluster, backups of components and networks in the IBM® Blockchain Platform are a matter of backing up its [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). These volumes are where ledgers and other types of storage are mounted so they can be used by nodes.

As a result, "backing up" a component or a network is the process of saving a copy of the relevant persistent volumes, while "restoring" a component or network involves bringing up components and pointing them to these saved volumes. The number of volumes that are associated with a node depends on the type of node. Peers have either one persistent volume (if using LevelDB) or two persistent volumes (if using CouchDB), while each node in the ordering service has a single persistent volume.

Because an ordering node cannot pull blocks from a peer, all peers must be backed up before the ordering nodes. This order ensures that the peers do not have any blocks on their ledgers that don't appear on the ledgers of the backed-up ordering nodes. If any peers have blocks that the ordering nodes do not have, the ledgers cannot be synchronized, making the restoration of the network impossible. If you are backing up your ledger as part of an upgrade to a new Fabric version, you do not have to synchronize this backup with the rest of the network. However, make sure to keep at least one ledger backup that is synchronized with the rest of the network in case a larger network restoration is necessary.

If you're using CouchDB, which is mounted in a separate container in the peer pod, back up the persistent volume associated with CouchDB before the volume associated with the peer pod. This is because the peer checks on CouchDB after booting and if anything is missing from CouchDB, the peer can push transactions to it which CouchDB integrates into the database. However, any transactions in the state database cannot be pushed back to the peer.

As a rule, it is a good practice to schedule backups (also known as "snapshots") to happen at the same time across the network.
Haproxy 8398d1 8895ad [CVE-2023-0361](https://nvd.nist.gov/vuln/detail/CVE-2023-0361). 
 Containerd 1.6.19 1.6.21 For more information, see the [change log](https://github.com/containerd/containerd/releases/tag/v1.6.21) and [security bulletin for CVE-2023-28642 and CVE-2023-27561](https://www.ibm.com/support/pages/node/7001317). 







 Change log for worker node fix pack 1.25.8_1539, released 29 March 2023 

The following table shows the changes that are in the worker node fix pack 1.25.8_1539. Worker node patch updates can be applied by updating, reloading (in classic infrastructure), or replacing (in VPC infrastructure) the worker node.



Changes since version 1.25.8_1537

 Component Previous Current Description 

 Ubuntu 20.04 packages 5.4.0-144 5.4.0-139 Downgrading kernel to address [Upstream canonical bug](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2009325). 







 Change log for master fix pack 1.25.8_1536, released 28 March 2023 

The following table shows the changes that are in the master fix pack 1.25.8_1536. Master patch updates are applied automatically.



Changes since version 1.25.6_1534

 Component Previous Current Description 

 Cluster health image v1.3.16 v1.3.17 Updated Go to version 1.19.7 and updated dependencies. 
 GPU device plug-in and installer 79a2232 a873e90 Updated Go to version 1.19.6. 
 IBM Calico extension 1308-amd64 1366-amd64 Updated to resolve [CVE-2023-23916](https://nvd.nist.gov/vuln/detail/CVE-2023-23916).
These nodes also run a small set of control plane agent services, as described in this table. 
 Bare metal nodes Bare metal nodes are delivered to you dedicatedly with VMware ESXi installed during provision. The bare metal nodes are equipped with SmartNIC and local storage to support VMware clusters on VPC. Your bare metal server API requests are routed to services deployed across control nodes in the region. All communication between control plane services occurs over a secure, encrypted network. 
 VPC control plane services Your VPC API requests are routed to services deployed across control nodes in the region. These services are replicated for availability and performance across multiple zones in the region whenever necessary. In addition to servicing API requests, these services monitor the region's hardware and capabilities, and perform orchestration to keep your VPC resources available and performing optimally. All communication between control plane services occurs over a secure, encrypted network. Since your instances do not run on control nodes, they are separated from these services. 
 VPC control plane agent services A small set of VPC control plane agents is deployed across hypervisor nodes in the region. For example, agents are used to create new virtual server instances on the nodes, and to forward logs, metrics, and alerts off the node for use by the broader VPC control plane services. All communication between control plane services occurs over a secure, encrypted network. 
 VPC control plane data store Your VPC resources are persisted in a data store that is replicated across all zones in the region. This store contains metadata about these resources only, and does not contain your data. For example, this store contains information about each VPC Image resource, such as its name, CRN, and creation date. However, it does not contain the image data itself, which is hosted on the storage devices. All communication between control plane services and the control plane data store is secure and encrypted. 
 Block Storage Each zone of each region contains a set of replicated storage devices, which host the data for your VPC volumes. Your data is always encrypted at rest, and is isolated to your account.
If any node is over 80% in either CPU or memory usage, consider provisioning more nodes or reducing your workload.

oc top node

Example output.

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
10.001.1.01 640m 16% 6194Mi 47%
10.002.2.02 2686m 68% 4024Mi 30%
10.003.3.03 2088m 53% 10735Mi 81%
2. Make sure that you have [removed any added webhooks](https://cloud.ibm.com/docs/containers?topic=containers-ts-delete-webhooks) from the cluster.







 Gathering data 

Follow the steps to gather the relevant worker node data.



1. Get the details of each node. Save the output details to include in your support ticket.

oc describe node <node-ip-address>
2. Run the [Diagnostics and Debug Tool](https://cloud.ibm.com/docs/containers?topic=containers-debug-tool) and the [Openshift Diagnostics and Debug Tool](https://cloud.ibm.com/docs/openshift?topic=openshift-debug-tool). Export the kube and network test results to a compressed file and save the file to include in your support ticket. If all workers in your cluster are affected, you can skip this step as the debug tools cannot work properly if all worker nodes are disrupted.
3. Show that there are no added mutating or validating webhooks remaining in your cluster by getting the webhook details. Save the command output to include in your support ticket. Note that the following mutating webhooks might remain and do not need to be deleted: alertmanagerconfigs.openshift, managed-storage-validation-webhooks, multus.openshift.io, performance-addon-operator, prometheusrules.openshift.io,snapshot.storage.k8s.io.
: IBM manages the [node tuning operator](https://docs.openshift.com/container-platform/4.11/scalability_and_performance/using-node-tuning-operator.html), which runs a daemon set on each worker node in the cluster to tune worker nodes. You can apply custom profiles to your worker nodes to improve performance. For more information, see [Modifying worker node settings by using the Node Tuning Operator](https://cloud.ibm.com/docs/openshift?topic=openshift-kernelworker-nto).

openshift-cluster-samples-operator
: The [samples operator](https://docs.openshift.com/container-platform/4.11/openshift_images/configuring-samples-operator.html) manages select image streams and templates that come with the Red Hat OpenShift cluster by default. You can deploy these templates from the [Developer perspective in the Red Hat OpenShift web console](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy_appopenshift_console).

openshift-cluster-storage-operator
: The cluster storage operator makes sure that a default storage class is set.

openshift-console, openshift-console-operator
: The Red Hat OpenShift web console is a user-friendly, web-based interface that you can use to manage the Red Hat OpenShift and Kubernetes resources that run in your cluster. You can also use the console to display an oc login token to authenticate to your cluster from a CLI. For more information, see [Navigating the Red Hat OpenShift console](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_tutorialopenshift_oc_console).

openshift-dns, openshift-dns-operator
: The DNS project includes the components to validate incoming network traffic against the iptables rules that are set up on the worker node, and proxies requests that are allowed to enter or leave the cluster.
If your apps require more resources than you have available on your worker node, use a different worker node flavor that can fulfill these requirements. You know that your worker nodes are overloaded when they frequently report back a status of NotReady or evict pods due to the lack of memory or other resources.
* Larger versus smaller worker node flavors: Larger nodes can be more cost efficient than smaller nodes, particularly for workloads that are designed to gain efficiency when they process on a high-performance machine. However, if a large worker node goes down, you need to be sure that your cluster has enough capacity to safely reschedule all the workload pods onto other worker nodes in the cluster. Smaller worker can help you scale safely.
* Replicas of your app: To determine the number of worker nodes that you want, you can also consider how many replicas of your app that you want to run. For example, if you know that your workload requires 32 CPU cores, and you plan to run 16 replicas of your app, each replica pod needs 2 CPU cores. If you want to run only one app pod per worker node, you can order an appropriate number of worker nodes for your cluster type to support this configuration.









 How do I monitor resource usage and capacity in my cluster? 

Now that you have a good estimate of your app size and the worker nodes that you need to run your workload, you can deploy your app and continuously monitor performance to maximize your compute resource usage.



1. [Create your cluster](https://cloud.ibm.com/docs/containers?topic=containers-clusters) with the worker pool flavor and number of worker nodes that you [previously estimated](https://cloud.ibm.com/docs/containers?topic=containers-strategysizing_workers).
2. Review what compute resources your cluster uses by default and calculate the remaining cluster capacity that you can use for your workloads.



1. With the IBM Cloud IAM Manager service access role for the cluster in all namespaces: [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)
2.
ibmcloud oc worker-pool taint set --worker-pool <WORKER_POOL> --cluster <CLUSTER> --taint logging-monitoring=node:NoExecute







 Step 3: Configure the Red Hat OpenShift on IBM Cloud monitoring stack 



1. For conceptual information about the monitoring stack, see [Understanding the monitoring stack](https://docs.openshift.com/container-platform/4.6/monitoring/understanding-the-monitoring-stack.html).
2. To configure the monitoring stack with supported options, see [Maintenance and support for monitoring](https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.htmlmaintenance-and-support_configuring-the-monitoring-stack).



1. Create and edit the cluster-monitoring-config ConfigMap in the openshift-monitoring project with the following configuration. For more information, see the [Configuring the monitoring stack](https://docs.openshift.com/container-platform/4.6/monitoring/configuring-the-monitoring-stack.htmlconfiguring-the-monitoring-stack_configuring-the-monitoring-stack). The sample code configures the monitoring stack to complete the following tasks:





* Run the monitoring stack on the provisioned worker pool.
* Add a retention period of 1 year.
* Add a 100 GB persistent volume to Prometheus.



data:
config.yaml: 
enableUserWorkload: true
prometheusOperator:
tolerations:
- key: "logging-monitoring"
operator: "Equal"
value: "node"
effect: "NoExecute"
prometheusK8s:
retention: 1y
volumeClaimTemplate:
spec:
storageClassName: ibmc-vpc-block-retain-general-purpose
volumeMode: Filesystem
resources:
requests:
storage: 100Gi
tolerations:
When you update your worker nodes, the ConfigMap helps prevent downtime for your apps as the apps are rescheduled orderly onto available worker nodes.
4. Update all the worker nodes in your cluster to remove the public cloud service endpoint configuration.

ibmcloud oc worker update --cluster <cluster_name_or_ID> --worker <worker1,worker2>

By issuing the update command, the worker nodes are reloaded to pick up the service endpoint configuration. If no worker update is available, you must reload the worker nodes manually with the ibmcloud oc worker reload[command](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload). If you reload, be sure to cordon, drain, and manage the order to control the maximum number of worker nodes that are unavailable at a time.









 Changing your worker node VLAN connections 

When you create a cluster, you choose the public and private VLANs to connect your worker nodes to. Your worker nodes are part of worker pools, which store networking metadata that includes the VLANs to use to provision future worker nodes in the pool. You might want to change your cluster's VLAN connectivity setup later, such as if the worker pool VLANs in a zone run out of capacity, and you need to provision a new VLAN for your cluster worker nodes to use.

Trying to change the service endpoint for master-worker communication instead? Check out the topics to set up the [private](https://cloud.ibm.com/docs/openshift?topic=openshift-cs_network_clusterset-up-private-se) service endpoint.

Before you begin [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).

To change the VLANs that a worker pool uses to provision worker nodes.



1. List the names of the worker pools in your cluster.

ibmcloud oc worker-pool ls --cluster <cluster_name_or_ID>
2.
There is no charge for this tooling or the Kubernetes cluster on IBM Cloud where it runs. 





 OpenShift cluster Description 

 Operator A Kubernetes operator that is used to deploy the console. 
 Routes An OpenShift route is a way to expose a service by giving it an externally reachable hostname. 
 Proxy The IBM Blockchain Platform proxy is responsible for routing traffic to the correct peer, CA and ordering nodes by using host header routing. 
 Peers, CAs, Ordering nodes These are the nodes that are created. Note: these nodes could also be imported from other Kubernetes Cluster Service Instances. Because the keys are never stored by IBM, every peer and ordering node includes a gRPC web proxy that allows the console to communicate with each node by using the keys in the wallet. 
 RBAC Role based access control. The IBM Blockchain Platform configures [OpenShift RBAC](https://docs.openshift.com/container-platform/4.3/authentication/using-rbac.html) in the cluster which is required to manage blockchain components in the cluster. 









 Integrating with IBM Cloud and other third-party services 

IBM Blockchain Platform can leverage a suite of services provided in the IBM Cloud catalog to enable users more visibility into their network or to integrate with other services.

Zoom

![IBM Blockchain Platform integrations](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/integrations-updated.png)

Figure 2.IBM Cloud Integrations

Access control



* Securely authenticate users and control access to all cloud resources using [IBM Cloud Identity and Access Management (IAM)](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-work_iam).



Monitoring
You can use and manage the cross-regional snapshot in the target region independently from the parent volume or the original snapshot.

Creating a cross-regional copy affects billing. You're charged for the data transfer and the storage consumption in the target region separately.

You can create and manage cross-regional snapshot copies in the UI, from the CLI, with the API, or Terraform.





 Tags for Block Storage for VPC snapshots 

You can apply [user tags](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-aboutsnapshots-about-user-tags) and [access management tags](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-aboutsnapshots-about-mgt-tags) to snapshots of your Block Storage for VPC volumes to better control and organize them across the VPC.



 User tags for snapshots 

User tags are uniquely identified by a Cloud Resource Name (CRN) identifier. When you create a user tag, you provide a unique name within your billing account. You can define user tags in label or key-value format. You can create user tags or add existing tags to snapshots. You can also add tags when you use a snapshot of a volume during [instance creation](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-manage&interface=uisnapshots-vpc-instancevol-tags-ui).