vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Using certificates from an external Certificate Authority 

In this tutorial, you learn how to use certificates that were generated by an external Certificate Authority (CA) with your IBM® Blockchain Platform network. After you gather the required certificates for a peer or ordering node, you build a Membership Service Provider (MSP) definition that is used by your blockchain components.

While it is always possible to use a CA that you create with the console to generate identities, you can also use certificates that were generated by a CA external to your network, one that is not hosted by IBM, and then build the required organization Membership Service Provider (MSP) definition that is used by your blockchain components.



 Objectives 



* Gather certificates.
* Build MSP definition.
* Import MSP into the console.
* Create and import the organization admin identity to the wallet.
* Deploy a blockchain node.







 Before you begin 



* [Install the IBM® Blockchain Platform](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-deploy-iksibp-v2-deploy-iks-deploy-process).







 Step 1: Gather certificates 

You need to request the following set of certificates from the external CA in [X.509](https://hyperledger-fabric.readthedocs.io/en/release-2.2/identity/identity.htmldigital-certificates) format. When you request the certificates from your provider, the core requirement for the certificates is that they are ECDSA with Curve P256 and that the private key uses the PKCS 8 standard. Also when you request the certificates, you need to specify a Node Organizational Unit (OU) attribute for some certificates as indicated in the Node OU column in the following tables. Certificates can be in the format of a .pem file or a base64 encoded string.

When you create a peer or ordering node, you need to provide the following four certificates in PEM format. When you request these certificates from the third-party CA, you need to request a specific Node Organizational Unit (OU) attribute for some certificates as indicated in the Node OU column.



Table 1. Node certificates
: If you create a standard classic cluster, you can choose to provision your worker nodes on bare metal physical servers (instead of virtual server instances). With bare metal machines, you have additional control over the compute host, such as the memory or CPU. This setup eliminates the virtual machine hypervisor that allocates physical resources to virtual machines that run on the host. Instead, all a bare metal machine's resources are dedicated exclusively to the worker, so you don't need to worry about "noisy neighbors" sharing resources or slowing down performance. Bare metal servers are dedicated to you, with all its resources available for cluster usage.

Encrypted disks
: By default, every worker node is provisioned with two local SSD, AES 256-bit encrypted data partitions. The first partition contains the kernel image that is used to boot the worker node and is not encrypted. The second partition holds the container file system and is unlocked by using LUKS encryption keys. Each worker node in a cluster has its own unique LUKS encryption key, managed by IBM Cloud Kubernetes Service. When you create a cluster or add a worker node to an existing cluster, the keys are pulled securely and then discarded after the encrypted disk is unlocked. Encryption can impact disk I/O performance. For workloads that require high-performance disk I/O, test a cluster with encryption both enabled and disabled to help you decide whether to turn off encryption.

Expert AppArmor policies
: Every worker node is set up with security and access policies that are enforced by [AppArmor](https://wiki.ubuntu.com/AppArmor) profiles that are loaded into the worker node during bootstrapping. AppArmor profiles can't be changed by the user or owner of the machine.

SSH disabled
: By default, SSH access is disabled on the worker node to protect your cluster from malicious attacks. When SSH access is disabled, access to the cluster is forced via the Kubernetes API server. The Kubernetes API server requires every request to be checked against the policies that are set in the authentication, authorization, and admission control module before the request is executed in the cluster.
* stop: Stops the service by scaling down the replicas in an order that limits possible dependency issues. This option also stores an annotation within the deployment or statefulset. If you subsequently try to use the script to start the service and the annotation doesn't exist in one of the deployments or statefulsets, the script fails.
* start: You can only use this command to start the service if the wactl script was used to stop the service.
* restart: Use this method to scale down the replicas, but to keep at least one replica for each pod running at all times to prevent a disruption in service.
* clean: Removes the annotation that was created by the stop option from all objects including datastores.



* release: The release name is hardcoded as watson-assistant.
* cli: Specify the command line interface you are using.

Specify oc for OpenShift and kubectl for Kubernetes.
* include-ds: Optional. Indicates that you want to perform the action on the data sources. When you specify this parameter with the restart action, all of the data sources are restarted except Redis.







 To manually scale the cluster all the way down and back 

To scale down the cluster all the way, you must scale down the deployed services in the following order:



* ui deployment
* store deployment
* gateway deployment
* All other deployments
* All statefulsets





1. If you have local storage, complete the following steps:



* Connect over SSH to each worker node, and then run the following command from a given worker to synchronize data to the same directory on each other worker.

The destination folders should be empty. If they're not, empty them.

rsync -av /mnt/local-storage/storage/watson/assistant/{yourPV} {other worker}:/mnt/local-storage/storage/watson/assistant
* Double-check the postgres-keeper persistent volume permissions on each worker node.

You might need to scale postgres-keeper deployments first to see which persistent volumes they claim.
If you delete the worker nodes or add worker nodes to the cluster, you must update your allowlist accordingly.

ibmcloud ks worker ls --cluster <cluster_name_or_ID>



2. Add the subnet CIDRs or individual worker node IP addresses to your service's allowlist or your on-premises allowlist for outbound traffic.
3. Repeat these steps for each cluster that you want to allow traffic to or from.









 Updating IAM allowlists for Kubernetes Service IP addresses 

By default, all IP addresses can be used to log in to the IBM Cloud console and perform actions to manage your cluster, such as creating, updating, deleting or viewing credentials. In the IBM Cloud Identity and Access Management (IAM) console, you can [create an allowlist by specifying which IP addresses have access](https://cloud.ibm.com/docs/account?topic=account-ips), and all other IP addresses are restricted.

In your allowlist, you must also add the CIDRs of the IBM Cloud Kubernetes Service control plane for the zones in the region where your cluster is located so that IBM Cloud Kubernetes Service can create or access the following components:



* Ingress ALBs
* VPC LoadBalancers
* VPC security groups
* VPE gateways



Before you begin, the following steps require you to change the IAM allowlist for the user whose credentials are used for the cluster's region and resource group infrastructure permissions. If you are the credentials owner, you can change your own IAM allowlist settings. If you are not the credentials owner, but you are assigned the Editor or Administrator IBM Cloud IAM platform access role for the [User Management service](https://cloud.ibm.com/docs/account?topic=account-account-services), you can update the restricted IP addresses for the credentials owner.



1. Identify what user credentials are used for the cluster's region and resource group infrastructure permissions.



1. Check the API key for a region and resource group of the cluster.

ibmcloud ks api-key info --cluster <cluster_name_or_ID>

Example output
Classic clusters: Why does source IP preservation fail when using tainted nodes? 

Classic infrastructure

 What’s happening 

You [enabled source IP preservation for an Ingress ALB](https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotationspreserve_source_ip_classic) service by changing externalTrafficPolicy to Local in the service's configuration file. However, no traffic reaches the back-end service for your app.

 Why it’s happening 

When you enable source IP preservation for Ingress ALB services, the source IP address of the client request is preserved. The service forwards traffic to app pods on the same worker node only to ensure that the request packet's IP address isn't changed.

Typically, Ingress ALB service pods are deployed to the same worker nodes that the app pods are deployed to. However, some situations exist where the service pods and app pods might not be scheduled onto the same worker node. If you use [Kubernetes taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) on worker nodes, any pods that don't have a taint toleration are prevented from running on the tainted worker nodes. Source IP preservation might not be working based on the type of taint you used:



* Edge node taints: You [added the dedicated=edge label](https://cloud.ibm.com/docs/containers?topic=containers-edgeedge_nodes) to two or more worker nodes on each public VLAN in your cluster to ensure that Ingress pods deploy to those worker nodes only. Then, you also [tainted those edge nodes](https://cloud.ibm.com/docs/containers?topic=containers-edgeedge_workloads) to prevent any other workloads from running on edge nodes. However, you didn't add an edge node affinity rule and toleration to your app deployment. Your app pods can't be scheduled on the same tainted nodes as the service pods, and no traffic reaches the back-end service for your app.
* Individual worker node IP addresses: If you have a small number of worker nodes that run only one app and don't need to scale, or if you want to add only one worker node, list all the worker nodes in your cluster and note the Public IP addresses. If your worker nodes are connected to a private network only and you want to connect to IBM Cloud services by using the private cloud service endpoint, note the Private IP addresses instead. Only these worker nodes are added. If you delete the worker nodes or add worker nodes to the cluster, you must update your allowlist accordingly.

ibmcloud oc worker ls --cluster <cluster_name_or_ID>



3. Add the subnet CIDR or IP addresses to your service's allowlist for outbound traffic or your on-premises allowlist for inbound traffic.
4. Repeat these steps for each cluster that you want to allow traffic to or from.







 Updating IAM allowlists for Kubernetes Service IP addresses 

By default, all IP addresses can be used to log in to the IBM Cloud console and perform actions to manage your cluster, such as creating, updating, deleting or viewing credentials. In the IBM Cloud Identity and Access Management (IAM) console, you can [create an allowlist by specifying which IP addresses have access](https://cloud.ibm.com/docs/account?topic=account-ips), and all other IP addresses are restricted.

In your allowlist, you must also add the CIDRs of the Red Hat OpenShift on IBM Cloud control plane for the zones in the region where your cluster is located so that Red Hat OpenShift on IBM Cloud can create or access components such as Ingress ALBs or the Red Hat OpenShift web console, which requires all control plane IP addresses.

Before you begin, the following steps require you to change the IAM allowlist for the user whose credentials are used for the cluster's region and resource group infrastructure permissions. If you are the credentials owner, you can change your own IAM allowlist settings.
By default IBM Cloud Kubernetes service clusters are preconfigured with a Calico network plug-in that secures the public network interface of every worker node in the cluster. By configuring Kubernetes and Calico network policies you can easily control the inbound and outbound network traffic. For more information, see [Controlling traffic with network policies](https://cloud.ibm.com/docs/containers?topic=containers-network_policies). In addition, you can also configure Edge worker nodes on your IBM Cloud Kubernetes service clusters to restrict the worker nodes that are exposed to the public internet. See [Restricting network traffic to edge worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-edge) to learn more. If your configuration includes a network firewall, there are ports that need to be exposed to allow network traffic through. The following section describes how to find the ports that can be exposed and what they are for.







 Internet Ports 

IBM Blockchain Platform exposes certain ports associated with each node type that are used by client applications, for example, when sending transactions to peers or the ordering service. If you have configured a firewall, you will need to expose these ports in your network for the transactions to reach their destination and for the nodes to be able to respond to the requests. IBM Blockchain Platform supports the HTTPS IP Security protocol.

If your IBM Blockchain Platform instance is linked to an OpenShift cluster in IBM Cloud, port 443 needs to be exposed.

If your IBM Blockchain Platform instance is linked to an IBM Cloud Kubernetes service cluster, use the following table to identify the ports that need to be exposed:



Table 2. Ports on IBM Cloud

 Port Number Description 

 CA ports 
 Fabric CA 7054 Port 7054 is exposed from the CA container. This is exposed all the way to the customer application via the service. 
 Peer ports 
 Peer gRPC API 7051 This is the port used by the clients to communicate with the peer. 
 Smart contract containers 7052 This is the port used by the smart contract containers to communicate with the peer.
Planning your Portworx setup 

Before you create your cluster and install Portworx, review the following planning steps.



* Decide where you want to store the Portworx metadata. You can use KVDB or an external Database instance. For more information, see [Understanding the key-value store](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_portworx_kv_store). To learn more about what the key value does, see the [Portworx documentation](https://docs.portworx.com/concepts/internal-kvdb/).
* Decide whether you want encryption. You can use Hyper Protect Crypto Services or IBM Key Protect. For more information, see [Understanding encryption for Portworx]/docs/openshift?topic=openshift-storage_portworx_encryption.
* Decide whether you want to use journal devices. Journal devices allow Portworx to write logs directly to a local disk on your worker node.
* VPC or Satellite clusters only - Decide whether you want to use cloud drives. Cloud drives allow you to dynamically provision the Portworx volumes. If you don’t want to use cloud drives, you must manually attach volumes to worker nodes.
* Review the [Limitations](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_portworx_planportworx_limitations).





 Limitations 

Review the following Portworx limitations.



Portworx limitations

 Limitation Description 

 Classic clusters Pod restart required when adding worker nodes. Because Portworx runs as a DaemonSet in your cluster, existing worker nodes are automatically inspected for raw block storage and added to the Portworx data layer when you deploy Portworx. If you add or update worker nodes to your cluster and add raw block storage to those workers, restart the Portworx pods on the new or updated worker nodes so that your storage volumes are detected by the DaemonSet. 
 VPC clusters Storage volume reattachment required when updating worker nodes.
Planning your Portworx setup 

Before you create your cluster and install Portworx, review the following planning steps.



* Decide where you want to store the Portworx metadata. You can use KVDB or an external Database instance. For more information, see [Understanding the key-value store](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_store). To learn more about what the key value does, see the [Portworx documentation](https://docs.portworx.com/concepts/internal-kvdb/).
* Decide whether you want encryption. You can use Hyper Protect Crypto Services or IBM Key Protect. For more information, see [Understanding encryption for Portworx]/docs/containers?topic=containers-storage_portworx_encryption.
* Decide whether you want to use journal devices. Journal devices allow Portworx to write logs directly to a local disk on your worker node.
* VPC or Satellite clusters only - Decide whether you want to use cloud drives. Cloud drives allow you to dynamically provision the Portworx volumes. If you don’t want to use cloud drives, you must manually attach volumes to worker nodes.
* Review the [Limitations](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_planportworx_limitations).





 Limitations 

Review the following Portworx limitations.



Portworx limitations

 Limitation Description 

 Classic clusters Pod restart required when adding worker nodes. Because Portworx runs as a DaemonSet in your cluster, existing worker nodes are automatically inspected for raw block storage and added to the Portworx data layer when you deploy Portworx. If you add or update worker nodes to your cluster and add raw block storage to those workers, restart the Portworx pods on the new or updated worker nodes so that your storage volumes are detected by the DaemonSet. 
 VPC clusters Storage volume reattachment required when updating worker nodes.
kind: Pod
metadata:
name: debug-${NODE}
namespace: default
spec:
containers:
- args: ["-c", "apk add tcpdump mtr curl; sleep 1d"]
command: ["/bin/sh"]
image: icr.io/armada-master/alpine:latest
imagePullPolicy: IfNotPresent
name: debug
resources: {}
securityContext:
privileged: true
runAsUser: 0
volumeMounts:
- mountPath: /host
name: host-volume
volumes:
- name: host-volume
hostPath:
path: /
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/hostname: ${NODE}
restartPolicy: Never
securityContext: {}
EOF
Show more
4. Log in to the debug pod. The pod's interactive shell is automatically opened. If the kubectl exec command fails, continue to option 3.

kubectl exec -it debug-${NODE} -- sh

You can use the kubectl cp command to get logs or other files from a worker node. The following example gets the /var/log/syslog file.

kubectl cp default/debug-${NODE}:/host/var/log/syslog ./syslog

Get the following logs to look for issues on the worker node.

/var/log/syslog
/var/log/containerd.log
/var/log/kubelet.log
/var/log/kern.log
5. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as ip, ifconfig, nc, ping, and ps, are already available in the shell.
Worker node utilization is calculated as the sum of the CPU and memory resources that are requested by all pods that run on the worker node, divided by the worker node resource capacity. The default value is 0.5.

scanInterval
: Set how often in minutes that the cluster autoscaler scans for workload usage that triggers scaling up or down. The default value is 1m.

expander
: Specify how the cluster autoscaler determines which worker pool to scale if you have multiple worker pools. The default value is random.
: random: Selects randomly between most-pods and least-waste.
: most-pods: Selects the worker pool that is able to schedule the most pods when scaling up. Use this method if you are using nodeSelector to make sure that pods land on specific worker nodes.
: least-waste: Selects the worker pool that has the least unused CPU after scaling up. If two worker pools use the same amount of CPU resources after scaling up, the worker pool with the least unused memory is selected.

ignoreDaemonSetsUtilization
: When set to true, the cluster autoscaler ignores DaemonSet pods when it calculates resource utilization for scale-down. The default value is false.

maxBulkSoftTaintCount
: Set the maximum number of worker nodes that can be tainted or untainted with PreferNoSchedule at the same time. To disable this feature, set to 0. The default value is 0.

maxBulkSoftTaintTime
: Set the maximum amount of time that worker nodes can be tainted or untainted with PreferNoSchedule at the same time. The default value is 10m.

skipNodes.withLocalStorage
: When set to true, worker nodes that have pods that are saving data to local storage are not scaled down. The default value is true.

skipNodes.withSystemPods
: When set to true, worker nodes that have kube-system pods are not scaled down. Do not set the value to false because scaling down kube-system pods might have unexpected results. The default value is true.

image.repository
: Specify the cluster autoscaler Docker image to use.
Use the ibmcloud oc worker update[command](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_update) regularly (such as monthly) to deploy updates and security patches to the operating system and to update the Red Hat OpenShift version that your worker nodes run. When updates are available, you are notified when you view information about the master and worker nodes in the IBM Cloud console or CLI, such as with the ibmcloud oc clusters ls or ibmcloud oc workers ls --cluster <cluster_name> commands. Worker node updates are provided by IBM as a full worker node image that includes the latest security patches. To apply the updates, the worker node must be reimaged and reloaded with the new image. Keys for the root user are automatically rotated when the worker node is reloaded.





 How does my worker node setup look? 

The following image shows the components that are set up for every worker node to protect your worker node from malicious attacks.

The image does not include components that ensure secure end-to-end communication to and from the worker node. For more information, see [network security](https://cloud.ibm.com/docs/openshift?topic=openshift-securitynetwork).

Zoom

![Worker node setup in Red Hat OpenShift on IBM Cloud excluding network security.](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/oc_worker_setup.png)

Figure 2. Worker node setup in Red Hat OpenShift on IBM Cloud excluding network security

CIS-compliant RHEL image
: Every worker node is set up with a Red Hat Enterprise Linux operating system that implements the benchmarks that are published by the Center of Internet Security (CIS). The user or the owner of the machine can't change this operating system to another operating system. To review the current RHEL version, run oc get nodes -o wide.