vania-janet/multiturn-rag-retrieval-data huggingface dataset:
When data comes in, it gets written in bulk to Cloudant.
5. redisConsumer.js: This script polls Event Streams at regular intervals for data. When data comes in, it gets written into a Redis "hash" where the hash key is the truck ID and the value is the truck data (lat/long). This hash arrangement allows all the truck data to be pulled down by the server.js script in one go. The great thing is that if for some reason one of your consumers crashes, it will pick up where it left off without losing any data, because Event Streams knows the last data point it served to each consumer. Decoupling data-producing systems from data-consuming systems in this way is a good design practice that increases resilience and fault tolerance.







 Summary 

In this tutorial, we have combined a number of IBM services to model a realistic IoT pattern where Internet-enabled devices on trucks are producing data that gets consumed by various applications for different purposes.

Because of the managed nature of all these services, as an application developer, you can focus on solving your business problems and leave us to do the heavy lifting of managing the infrastructure for you, thereby saving potentially months of work on non-core activities like server provisioning, software deployment, patching, security and others.

If you followed this tutorial, remember to de-provision your infrastructure to stop incurring charges. On your terminal, do the following:

cd terraform/
terraform destroy

If you want to take the next step in your developer journey, check out some of our [trial offers](https://www.ibm.com/products?types%5B0%5D=trial).
To use the 2.x smart contract lifecycle, an organization must have an endorsement policy defined. If any organization in the consortium (the list of organizations maintained by the ordering service that are allowed to create channels) do not have an endorsement policy defined, a warning message will appear on the Details page of the ordering service with a list of organization MSPs that must be updated.

The best practice to add this endorsement policy to the MSP is to delete the MSP from the system channel and then re-add the MSP. The console detects the fact that the MSP does not contain the endorsement policy and automatically adds it. Note that this action can only be completed by an ordering service administrator. You do not need to delete and re-add the MSPs in the configuration of any application channels that have already been created. For these MSPs, the endorsement policy is added as part of the process of deploying the smart contract.





 Create channel and join peers 

Because the lifecycle of a smart contract begins by installing the smart contract on a peer that is joined to a channel, you need to create a channel and include the organization members. The channel must be configured to run with the [Fabric 2.0 application capability](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-governibp-console-govern-capabilities-application-channels) level. Second, you need to deploy at least one peer (that must be running the Fabric v2.x image) for your organization and join it to the channel. If you haven't already created a channel and joined at least one peer to it, see the [Build a network](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-build-networkibp-console-build-network-structure) tutorial for instructions.

If you are creating a new channel, ensure that you select Advanced channel configuration so that you have the opportunity to specify the Channel Application Capability 2.0.0 or higher.





 Export and Import Membership Service Providers (MSPs)
The following message is displayed the first time to you connect after the update because there is a new SSH host key.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:+AuFsuEHkEpIZs0Z2XSvrEPDXH4bKJWVoKhD9gZOWm8.
Please contact your system administrator.
Add correct host key in /Users/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/username/.ssh/known_hosts:5
ECDSA host key for 52.116.29.107 has changed and you have requested strict checking.
Host key verification failed.

To update your image, run:

ibmcloud hpvs instance-update (NAME | CRN)

For information about the update command, see [hpvs-update-instance](https://cloud.ibm.com/docs/hpvs-cli-pluginhpvsinstanceupdate).



 Updating the authorized_keys file 

Before you update your IBM-provided image, make sure that the root users' authorized_keys file is located within /data/.ssh and the correct file permissions are in place.

If you update your virtual server and the file is not in the correct directory, you lose access to your virtual server forever as recovery is not possible.
Before you create a key ring for your Hyper Protect Crypto Services instance, keep in mind the following considerations:



* Every Hyper Protect Crypto Services instance comes with a default key ring.

Each newly created Hyper Protect Crypto Services instance comes with a generated key ring with an ID of default. All keys that are not associated with a specified key ring exist within the default key ring.
* Key rings can hold root keys and standard keys, but not EP11 keys.

Key rings can contain both root and standard keys. There is no limit on how many keys can exist within a key ring. Key rings don't apply to Enterprise PKCS #11 (EP11) keys.
* A key only can belong to one key ring at a time.

A key can belong to only one key ring. Key ring assignment happens upon key creation. If a key ring ID is not passed in upon creation, the key will belong to the default key ring. You can update the key ring after the key creation.
* You can create up to five keystores in a service instance for free, including key rings and EP11 keystores. The maximum number of key rings for a service instance is 50.

Each additional key ring or EP11 keystore is charged with a tiered pricing starting at $225 USD per month. For more information about pricing, see [the pricing sample](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-faq-pricingfaq-how-charge-hpcs).





 Creating key rings 

Before you can group keys into a key ring, you need to create a key ring first. You can use either the console or the key management service API to create a key ring.

You can create up to five keystores in a service instance for free, including key rings and EP11 keystores. Each additional key ring or EP11 keystore is charged with a tiered pricing starting at $225 USD per month.



 Creating key rings with the console 

Create a key ring with the console by completing the following steps:



1. [Log in to the IBM Cloud console](https://cloud.ibm.com/login).
2.
Updating a virtual server 

You can update the OCI Image that is used for your Hyper Protect virtual server.

To trigger an update, you must have the IAM editor role (platform access) on the server. You can only update the server by way of the Hyper Protect Virtual Servers [CLI](https://cloud.ibm.com/docs/hpvs-cli-plugin). Before you update a server, back up all your data and make sure you can restore the data from your backups. IBM can't restore any data for you.



 Updating the IBM-provided image 

The IBM-provided Ubuntu image is generally updated thru the standard Ubuntu package manager. Use the Ubuntu image provided to replace everything within your server with the new version. The Ubuntu image resets everything to the state of a new server except for the content within /data.

Older instances did set the public SSH key in a way, which is not compatible with the image update. Check the information in [Updating the authorized_keys file](https://cloud.ibm.com/docs/services/hp-virtual-servers?topic=hp-virtual-servers-update_vsupdate_authkey) to ensure that your configuration is compatible with the image update. Otherwise, you irretrievably lose access to the virtual server.

When you update your image, the server settings are updated, for example, the [hardening configuration](https://cloud.ibm.com/docs/hp-virtual-servers?topic=hp-virtual-servers-protect_vs). The update also creates a new SSH host key (the old key is deleted).

The following message is displayed the first time to you connect after the update because there is a new SSH host key.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
"access_token": "eyJhbGciOiJIUz......sgrKIi8hdFs",
"refresh_token": "SPrXw5tBE3......KBQ+luWQVY=",
"token_type": "Bearer",
"expires_in": 3600,
"expiration": 1473188353
}

For more information, see the [IAM Identity Services API](https://cloud.ibm.com/apidocs/iam-identity-token-apigettoken-apikey).

For more information, see the [Java SDK](https://github.com/IBM/java-sdk-core/blob/main/Authentication.md).

For more information, see the [SDK](https://github.com/IBM/ibm-cloud-sdk-common/blob/main/README.mdauthentication).

For more information, see the [Python SDK](https://github.com/IBM/python-sdk-core/blob/main/Authentication.md).

For more information, see the [Go SDK](https://github.com/IBM/go-sdk-core/blob/main/Authentication.md).

An IAM token is valid for up to 60 minutes, and it is subject to change. When a token expires, you must generate a new one. Use the property expires_in for the expiration of the IAM token that you have just created.
The Ansible playbooks include a set of modules that you can use to automate the deployment of your nodes, registering of identities, channel creation, and smart contracts installation and instantiation.



* Deploying your blockchain network After you deploy the IBM Blockchain Platform to a Kubernetes cluster on IBM Cloud, or on your supported Kubernetes distribution, there are three ways to build your network. To understand the benefit of using the Ansible playbooks, we compare the options:



1. Deploy your nodes and configure your network manually using the console user interface. This option is most useful for getting started and learning about the platform and how the console works. It requires stepping through multiple panels to configure all the settings for your nodes, organizations, channels, and smart contracts.
2. Use the IBM Blockchain Platform REST APIs to build your network. This option is for advanced users who want to leverage the available REST APIs, often in conjunction with the Hyperledger Fabric APIs, to set up and administer their blockchain network. Configuration is done through editing API parameters. Because the IBM Blockchain Platform APIs are mainly for create, retrieve, update, and delete operations on a node, you may need to also learn and use the Hyperledger Fabric APIs for some tasks.
3. Use the IBM Blockchain Platform Ansible Collection. After you are familiar with the process to set up the various nodes, organizations, channels, and smart contracts, you might want to automate the process. The Ansible playbooks provide a simple, straightforward, way to build and deploy a reproducible network. For getting started, a set of scripts is provided that stitches the relevant playbooks together into a single executable shell script of deployments to perform on your network. Configuration is done through editing .yml files. The Ansible playbooks also automate some of the tasks performed by the Fabric APIs, such as creating a channel, joining a peer to a channel, and installing and instantiating smart contracts. Even more, there are playbooks available for tearing down the configuration so you can reset your network as needed. This option is a great option for setting up networks for POCs, demos, development, and test networks.
Then locate the IP address for <prefix>-primary-0, which is the compute IP address that you need.
4. In addition to these two IP addresses, pick up the floating IP address that's attached to the login node.







 Setting up access 

You need to separately access both the storage and compute clusters.

This solution's automation always uses the same IP addresses, so there might be issues in the /.ssh/known_hosts file. If you encounter problems with this file, remove the offending entries from that file and make sure to get a clean SSH in place, otherwise the GUI access won't work.



 Accessing the storage cluster 



1. Open a new command line terminal.
2. Run the following command to access the storage cluster:

ssh -L 22443:localhost:443 root@{STORAGE_NODE_IP_ADDRESS} root@{FLOATING_IP_ADDRESS}

where STORAGE_NODE_IP_ADDRESS needs to be replaced with the storage IP address associated with <prefix>-scale-storage-0, which you gathered earlier, and FLOATING_IP_ADDRESS needs to be replaced with the floating IP address that you identified.
3. Open a browser on the local machine, and run [https://localhost:22443](https://localhost:22443). You will get an SSL self-assigned certificate warning with your browser the first time that you access this URL.
4. Enter your login credentials that you set up when you created your workspace to access the Spectrum Scale GUI.







 Accessing the compute cluster 



1. Open a new command line terminal.
2. Run the following command to access the compute cluster:

ssh -L 21443:localhost:443 root@{COMPUTE_NODE_IP_ADDRESS} root@{FLOATING_IP_ADDRESS}

where COMPUTE_NODE_IP_ADDRESS needs to be replaced with the storage IP address associated with <prefix>-primary-0, which you gathered earlier, and FLOATING_IP_ADDRESS needs to be replaced with the floating IP address that you identified.
3.
2. In the address bar, enter about:config.
3. Click Accept the Risk and Continue button.
4. In the Search preference name, type asyncc.
5. Select dom.events.asyncClipboard.readText and dom.events.testing.asyncClipboard by clicking the toggle icon on the right. The settings change from false to true.
6. Refresh the web console page.











 February 2023 



 New features and enhancements 



* Use time travel queries to retrieve and analyze historical data without having to develop extra application logic such as history tables. Netezza Performance Server time travel comes in handy when you want to track the history of data changes or reconstruct your data. By using this powerful tool, you can access historical data (data that was changed or deleted) at past points in time or within a past period of time. For more information, see [Getting started with time travel](https://cloud.ibm.com/docs/netezza?topic=netezza-introducing_tt).
* The nzbackup and nzrestore destination directory limit is updated. Previously, only 16 directory locations to read/write were supported. Now, the limit is 32.
* The location of postmaster and postgres core dumps is changed.





Table 1. New and previous locations of the postmaster and postgres core dumps.

 Core dump Previous location New location 

 Postmaster cores NZ_DATA_DIR/global NZ_LOG_DIR/postgres/postmaster/ 
 Postgres cores NZ_DATA_DIR/base NZ_LOG_DIR/postgres/postgres 





* Automatic pause and resume is enabled in the web console.







 Components 



* Netezza Performance Server 11.2.2.5
* Web console 4.0.11







 Known issues 

If a common table expression or derived table query contains column names or column aliases, which begin with an underscore, Netezza Performance Server deletes these columns in the query result set.
If there are no columns to display, Netezza Performance Server returns the following error.
Setting up the Portworx key-value store 



 Setting up a key-value store for Portworx metadata 

Decide on the key-value store that you want to use to store Portworx metadata.

Before you begin, review the [Planning your Portworx setup section](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_plan).

The Portworx key-value store serves as the single source of truth for your Portworx cluster. If the key-value store is not available, then you can't work with your Portworx cluster to access or store your data. Existing data is not changed or removed when the Portworx database is unavailable.

To set up your key-value store, choose between the following options.



* [Automatically set up a key-value database (KVDB) during the Portworx installation](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_store).
* [Set up a Databases for etcd service instance](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_kv_store).







 Using the Portworx key-value database 

Automatically set up a key-value database (KVDB) during the Portworx installation that uses the space on the additional local disks that are attached to your worker nodes.

You can keep the Portworx metadata inside your cluster and store it along with the operational data that you plan to store with Portworx by using the internal key-value database (KVDB) that is included in Portworx. For general information about the internal Portworx KVDB, see the [Portworx documentation](https://docs.portworx.com/concepts/internal-kvdb/).

To set up the internal Portworx KDVB, follow the steps in [Installing Portworx in your cluster](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_deploy).
Cloud Directory post-MFA flow



1. When a user successfully signs in to your application, they are prompted to enter their second authentication factor.
2. When the second authentication factor is completed successfully, two simultaneous actions occur:



1. App ID sends information about the sign-in to your configured extension.
2. The user is redirected to your application.





To configure a post-MFA extension:



1. Configure an extension point that can listen for a POST request. The endpoint must be able to read the payload that is sent by App ID. Optionally, it can also [decode and validate](https://cloud.ibm.com/docs/appid?topic=appid-token-validationlocal-validation) the JSON payload that is returned by App ID is not altered by a third party in any way. A string that is formatted as {"jws": "jws-format-string"} is returned that contains the following information:



Table 5. The information that App ID forwards to your extension point.

 Information Description 

 correlation_id A random number that is generated for each MFA session. If you have both a pre-mfa and a post-mfa extension, the number is the same for each. For example, 3bb9236c-792f-4cca-8ae1-ada754cc4555. 
 extension The name of your extension. For this use case, the extension is named postmfa. 
 status The MFA status. Options include: success and failed. 
 reason The reason for an MFA failure. For example, user locked out - exceeded maximum number of verification attempts. 
 device_type The type of device with which your user accesses your application. Options include: web, mobile. 
 source_ip The IP address of the device that makes the request to your app. For example, 127.0.0.1. 
 headers The information that is returned by the browser when a user attempts to sign in to your app.
The speed of your Ethernet connection must be faster than the expected maximum throughput from your volume. Generally, don't expect to saturate your Ethernet connection beyond 70% of the available bandwidth.

For example, if you have 6,000 IOPS and are using a 16-KB block size, the volume can handle approximately 94-MBps throughput. However, when you have a 1-Gbps Ethernet connection to your LUN, it becomes a bottleneck when your servers attempt to use the maximum available throughput. It's because 70 percent of the theoretical limit of a 1-Gbps Ethernet connection (125 MB per second) would allow for 88 MB per second only.







 Best practice 2 - Optimize the host and applications 



* Use the right I/O scheduler. I/O schedulers help to optimize disk access requests. They traditionally do achieve optimization by merging I/O requests. By grouping requests at similar sections of disk, the drive doesn't need to "seek" as often, improving the overall response time for disk operations. On modern Linux implementations, several I/O scheduler options are available. Each of these options has their own unique method of scheduling disk access requests.



* Deadline is the default I/O scheduler on Red Hat 7.9, and usually it does not need to be changed to a different I/O scheduler. It's latency-oriented scheduler and it works by creating a separate read queue and separate a write queue. Each I/O request has a timestamp that is associated with it to be used by the kernel for an expiration time. While this scheduler also attempts to service the queues based on the most efficient ordering possible, the expiration time acts as a "deadline" for each I/O request. When an I/O request reaches its deadline, it is pushed to the highest priority.
* No Operation (NOOP) is a basic scheduler that passes down the I/O that comes to it. This scheduler places all I/O requests into a FIFO (First in, First Out) queue. It's a useful tool for checking whether complex I/O scheduling decisions of other schedulers are causing I/O performance regressions.
Basic premise: Remote clients pass requests, including secure credentials, through a private server to COS.

Zoom

![reverse=proxy](https://cloud.ibm.com/docs-content/v1/content/d7287ec7418d843dbd7caf286d842c439df6979e/direct-link/images/reverse-proxy.png)

Figure 1. Reverse proxy

HTTPS (secure HTTP) COS requests are initiated from a client at a remote site. They're transmitted securely through IBM Cloud Direct Link, targeting one of a cluster of reverse proxy servers deployed in a customer’s IBM Cloud account. From there, requests are passed to a COS private endpoint, processed, and then the results returned to the remote calling client.

Any sample client code that works with COS should also work through a reverse proxy server. The only change that is required is that, instead of targeting a COS private endpoint URL published by IBM, the client targets the IP address or URL of the reverse proxy server.



 Installing your Nginx reverse proxy 

NginX is a mature, compact, and fast open source web server that excels at specialized tasks, including the reverse proxy server role.

The instructions and configuration information that follows (for setting up an NginX reverse proxy server) can work after you adapt it to your environment. If you get stuck or need additional information, see the reverse proxy portion of the [Nginx documentation](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) or search [stackoverflow](https://stackoverflow.com) for examples.



1. Provision your VSI or bare metal servers with minimal RHEL or CentOS Linux build (recommended).
2. For each VSI, enable the following security group rules on the public interface: allow_http, allow_https, allow_outbound, allow_ssh
3. For each VSI, enable allow_all and allow_outbound rules on the private interface; select Save.
4.