vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Custom](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-power-vscustom) Custom profiles are for nonproduct development for testing or development use only. These profiles are not intended for production deployments and are not supported or certified for SAP production. 
 [Small](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-power-vssmall) Small profiles are best for balanced workloads that require less CPU and storage consumption. 
 [Balanced](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-power-vsbalanced) Balanced profiles are best for midsize databases and common cloud applications with moderate traffic. 
 [Compute Intensive](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-power-vscomputeintensive) Compute Intensive profiles are best for moderate to high web traffic workloads. Compute profiles are best for workloads with intensive CPU demands, such as high web traffic workloads, production batch processing, and front-end web servers. 
 [Ultra Memory](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-power-vsumemory) Ultra High Memory profiles offer the highest vCPU to memory ratios to serve in-memory OLTP databases, such as SAP. 
 [High Memory](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-power-vshmemory) Very High Memory profiles are best for server OLAP databases, such as SAP NetWeaver. 







 Custom 

IBM Power Virtual Servers are available with fully adjustable CPU Cores and Memory (RAM GiB). When you create a virtual server instance, you can define a custom size of the IBM Power Virtual Server profile to use for nonproduction SAP HANA instances, in accordance with existing SAP HANA for IBM Power Systems best practices and guidance from SAP.

Custom profiles are for nonproduct development for testing or development use only.
[Balanced](https://cloud.ibm.com/docs/vpc?topic=vpc-vs-profilesvs-balanced) Balanced profiles offer a core to RAM ratio 1 vCPU to 4 GiB of RAM ratio and are best for midsize databases and common cloud applications with moderate traffic. 
 [Compute](https://cloud.ibm.com/docs/vpc?topic=vpc-vs-profilesvs-compute) Compute profiles offer a core to RAM ratio 1 vCPU to 2 GiB of RAM ratio and are best for moderate to high web traffic workloads. Compute profiles are best for workloads with intensive CPU demands, such as high web traffic workloads, production batch processing, and front-end web servers. 
 [Memory](https://cloud.ibm.com/docs/vpc?topic=vpc-vs-profilesvs-memory) Memory profiles offer a core to RAM ratio 1 vCPU to 8 GiB of RAM ratio and are best for memory caching and real-time analytics workloads. Memory profiles are best for memory intensive workloads, such as large caching workloads, intensive database applications, or in-memory analytics workloads. 



s390x processor architecture profiles can be used to provision IBM Cloud Hyper Protect Virtual Servers for IBM Cloud® Virtual Private Cloud, LinuxONE virtual server instances, and z/OS virtual server instances. These profiles are available in Japan (Tokyo), Brazil (São Paulo), Canada (Toronto), United Kingdom (London), and US East (Washington DC) regions.



 Balanced 

Balanced profiles provide a mix of performance and scalability for more common workloads with a ratio of 4 GiB of memory for every 1 vCPU of compute. The following table shows all Balanced profiles available for the IBM Z or LinuxONE (s390x architecture) processors.

Ensure that you select a secure execution enabled profile (for example, cz2e-2x4) when you enable the Run your workload with an OS and a profile protected by Secure Execution toggle (to activate support for secure execution images). Selecting any profile that is not secure execution enabled will cause the deployment of the virtual instance to fail.

s390x

s390x with secure execution



Table 2.
Now add the need to derive cross-user analytics. The way that you do that is to replicate all the user databases into a single analytics database. All good. This app has suddenly become successful, and the number of users grew in the range of 150 - 20,000. You have 20,000 replications just to keep the analytics database current. If you also want to run in an active-active disaster recovery setup, add another 20,000 replications, and the system stops functioning.

Instead, multiplex user data into fewer databases, or shard users into a set of databases or accounts, or both. That way, you do not need to replicate to provide an analytics database, but authentication becomes more complicated as IBM Cloudant provides only authentication at the database level.

It’s worth stating that the “database-per-user” approach is tempting because IBM Cloudant permissions are “per database”, but it’s not really the users’ fault that this pattern emerged.





 Avoid writing custom JavaScript reduce functions 

The MapReduce views in IBM Cloudant are awesome. However, with great power comes great responsibility. The map part of a MapReduce view is built incrementally, so shoddy code in the map impacts only indexing time, not query time. The reduce-part, unfortunately, runs at query time. IBM Cloudant provides a set of built-in reduce functions that are implemented internally in [Erlang](https://www.erlang.org/). These functions are performant at scale while your hand-crafted JavaScript reduces are not.

If you find yourself writing reduce functions, stop and consider whether you can reorganize your data so that writing reduce functions isn’t necessary. Or so that you’re able to rely on the built-in reducers.

Views on partitioned databases do not support custom reduces, which is one factor that contributes to the significant speed-up queries only such views can offer.

For more information, see IBM Cloudant docs on [reduces](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreducereduce-functions).





 Use time boxed databases for ever-growing data sets
"target": "https://mysecondaccount.cloudant.com/b"
}

The source and target do not need to be on the same account. The source and target database names do not need to match. You must be authorized to access both the source and target, and you must be authorized to write to the target.





 Is replication run on the source or the destination? 

Replication can be started at either the source or the destination end. This choice means that you can decide whether account A is pushing data to account B, or account B is pulling data from account A. In some cases, it might not be possible to run replication in either configuration, for example when one account is behind a firewall. Replication happens over HTTPS and so no non-standard ports need be opened. The decision as to which device starts replication is yours.





 How does replication affect the list of changes? 

You can get a list of changes made to a document by using the [_changes endpoint](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-databasesget-changes). However, the distributed nature of IBM Cloudant databases means that the response that is provided by the _changes feed cannot be a simple list of changes that occurred after a particular date and time.

The [CAP Theorem](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cap-theoremcap-theorem) discussion makes it clear that IBM Cloudant uses an "eventually consistent" model. This model means you might get different results when you ask two different replicas of a database for a document at the same time. This can happen when one of the database copies is still waiting to finish replication.

Eventually, the database copies complete their replication so that all the changes to a document are present in each copy.

This "eventual consistency" model has two characteristics that affect a list of changes:



1. A change that affects a document almost certainly takes place at different times in different copies of the database.
2. The order in which changes affect documents might differ between different copies of the database, depending on when and from where the replication took place.
Upgrading to a new Major Version 

When a major version of a database is at its end of life (EOL), upgrade to a current major version. Find the available versions of Redis on the [IBM Cloud® Databases for Redis the catalog](https://cloud.ibm.com/catalog/databases-for-redis) page, from the Cloud Databases CLI plug-in command [ibmcloud cdb deployables-show](https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-referencedeployables-show), or from the Cloud Databases API [/deployables](https://cloud.ibm.com/apidocs/cloud-databases-apiget-all-deployable-databases) endpoint.



 Simple upgrade path 

Since most uses of Databases for Redis serve as a cache where the data tends to be transient, it is best to create a new deployment. In this simple method, provision a new deployment using the latest version of Databases for Redis and then point your application to the new deployment. This will build up the cache directly through initial use.





 If you require data migration 

If you absolutely must retain your persisted data when upgrading to the next major version of Databases for Redis, take steps to successfully migrate all of your data. You need to copy all the key-value pairs into the new deployment. Do this from your old instance to your new deployment by using a method appropriate to your use case.

Databases for Redis has provided an unsupported example script to help simplify this process.



 The migration script 

Warning! This script is unsupported and exists as an example of what you can build to move Databases for Redis data from one instance to another as part of a version upgrade. All source code and/or binaries that are attached to this document are referred to here as "the Program". IBM is not providing program services of any kind for the Program. IBM is providing the Program on an "AS IS" basis without warranty of any kind.
It's a side effect of having disconnected copies of data that can be modified without locking. IBM Cloudant chooses to handle conflicts by not discarding clashing changes, but storing them as a conflict.

To check a document for conflicts, simply add ?conflicts=true to a fetch of a single document. Any conflicting revisions are listed in the _conflicts array.

Unwanted revisions can be removed by using the normal DELETE operation, specifying the rev token of the revision you want to delete. The bulk API is also good for removing conflicting revisions, even for removing multiple conflicts from the same document.

To summarize, IBM Cloudant is a distributed service that stores databases, which are broken into multiple shards, with three copies of each shard spread around a ring of storage nodes. IBM Cloudant is eventually consistent, favoring high availability over strong consistency.

Avoid writing to the same document over and over so as not to create conflicts. Although conflicts are sometimes inevitable in replication situations.

Embrace eventual consistency - don't try to make IBM Cloudant consistent.

IBM Cloudant products based on CouchDB 4 might have a different consistency model.

That's the end of this course. For more information, see the [IBM Cloudant documentation](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant) and [blog](https://blog.cloudant.com/).
The potential for rewinds makes the changes feed unsuitable for an application that expects queue-like behavior.

To reiterate, IBM Cloudant's changes feed promises to deliver a document at least one time in a changes feed, and gives no guarantees about repeated values across multiple requests.
* Does the changes feed operate in "real time"?

The changes feed doesn't guarantee how quickly an incoming change appears to a client that consumes the changes feed. Applications must not be developed with the assumption that data inserts, updates, and deletes are immediately propagated to a changes reader.
* Why don't all individual document changes appear in the changes feed?

If a document is updated several times in between changes feed calls, then the changes feed might reflect only the most recent of these changes. The client does not receive every change to every document.

The IBM Cloudant changes feed isn't a transaction log that contains every event that happened in time order.
* Can I use a filtered changes feed for operational queries?

Filtering the changes feed, and by extension, running filtered replication has its uses:



* Copying data from source to target but ignoring deleted documents.
* Copying data but without index definitions (design documents).



This [blog post](https://blog.cloudant.com/2019/12/13/Filtered-Replication.html) describes how supplying a selector during replication makes work of these use cases run smoothly.

The changes feed with an accompanying selector parameter is not the way to extract slices of data from the database on a routine basis. It must not be used as a means of running operational queries against a database. Filtered changes are slow (the filter is applied to every changed document in turn, without the help of an index). This process is much slower than creating a secondary index (such as a MapReduce view) and querying that view.
* Does a feed=continuous changes feed continue to run indefinitely?

No, IBM Cloudant does not guarantee connection duration for a continuous changes feed. It might be regularly disconnected by the server for any number of reasons, which include maintenance, security, or network errors.
See the following example response to the request:

{
"offset": 0,
"rows": [
{
"id": "abc125",
"key": "amelie.smith@aol.com",
"value":
"Amelie Smith",
true,
"2020-04-24T10:42:59.000Z"
]
},
{
"id": "abc123",
"key": "bob.smith@aol.com",
"value":
"Bob Smith",
true,
"2019-01-24T10:42:59.000Z"
]
}
],
"total_rows": 2
}
Show more







 Indexes 

When a view is defined in a design document, a corresponding index is also created, based on the information defined within the view. Use indexes to find documents by criteria other than their _id field. For example, you might select by a field, or combination of fields, or by a value that is computed by using the contents of the document. The index is populated as soon as the design document is created. On large databases, this process might take a while.

If one of the following events occurs, the index content is updated incrementally and automatically:



* A new document is added to the database.
* An existing document is deleted from the database.
* An existing document in the database is updated.



View indexes are rebuilt entirely when the view definition changes, or when another view definition in the same design document changes. The rebuild ensures that changes to the view definitions are reflected in the view indexes. To ensure that the rebuild happens, a 'fingerprint' of the view definition is created whenever the design document is updated. If the fingerprint changes, then the view indexes are rebuilt.

View index rebuilds occur when you change any one view from all the views that are defined in the design document. For example, if you have a design document with three views, and you update the design document, all three view indexes within the design document are rebuilt.
* UI
* CLI
* API







 Allowlisting 

Cloud Databases now supports context-based restrictions, which give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with context-based restrictions and identity and access management (IAM) policies. Context-based restrictions check that an access request comes from an allowed context that you configure. Since both IAM access and context-based restrictions enforce access, context-based restrictions offer protection even in the face of compromised or mismanaged credentials. For more information, see [Protecting Cloud Databases resources with context-based restrictions](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-cbr).

To restrict access to your databases, allowlist specific IP addresses or ranges of IP addresses on your deployment.

If you use allowlists in your environment, allowlist our services by using the list of subnets for each region.

While IBM values the use of inclusive language, terms that are outside of IBM's direct influence are sometimes required for the sake of maintaining user understanding. As other industry leaders join IBM in embracing the use of inclusive language, IBM continues to update terminology to reflect those changes. We updated documentation to reflect changes in terminology from whitelist to allowlist. You might encounter continued references to this former terminology while we work to implement these deeper changes to code, API, and CLI commands.



 Using IP allowlists on your Deployment 

When you create an allowlist, only IP addresses that match the allowlist or are in the range of IP addresses in the allowlist can connect to your deployment. Allowlists can be enabled for both public endpoints and private endpoints. When no IP addresses are in the allowlist, the allowlist is disabled and the deployment accepts connections from any IP address.

Each deployment is limited to 100 allowlist entries.

Even if not explicitly allowlisted, IBM Cloud management services are still able to connect.



 Setting an allowlist in the UI 

The UI for managing allowlists is on the Settings tab of your Deployment Overview.

Zoom

!
* UI
* CLI
* API







 Service Endpoints Integration 

All Cloud Databases deployments offer integration with [IBM Cloud Service Endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview) to enable connections to your deployments from the public internet and over the IBM Cloud Private network.

Service Endpoints are available in all IBM Cloud Multi-Zone Regions and some single-campus multizone regions. Deployments in all other regions are able to use Service Endpoints.



 Public Endpoints 

Public endpoints provide a connection to your deployment on the public network. At provision, a public endpoint is the default option for all deployments. Your environment needs to have internet access to connect to a deployment.

For enhanced security, it is recommended that users connect to their Cloud Databases deployments using private endpoints instead of public endpoints.





 Private Endpoints 

A deployment with a service endpoint on the private network gets an endpoint that is not accessible from the public internet. All traffic is routed to hardware dedicated to Cloud Databases deployments and remains on the IBM Cloud Private network. All traffic to and from this endpoint is free and unmetered on the condition that the traffic remains in IBM Cloud. After your environment has access to the IBM Cloud Private network, an internet connection is not required to connect to your deployment.

For more information, see [Secure access to services using service endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview).

Deployments with private endpoints are reachable from any account within the private network and access to each instance requires authentication. To restrict this access to specific IP addresses, or ranges of IP addresses, configure [Context-based restrictions](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-cbr).





 Enabling Service Endpoints 

To use connections over the public internet, you do not have to enable Service Endpoints on your IBM Cloud account.
hyperp-dbaas-mongodb.cluster.delete Delete a database cluster 
 hyperp-dbaas-mongodb.log.get Download a log file 
 hyperp-dbaas-mongodb.logging.enable Enable logging for the cluster 
 hyperp-dbaas-mongodb.monitoring.enable Enable monitoring for the cluster 
 hyperp-dbaas-mongodb.cluster.resource.scale Scale cluster resources 
 hyperp-dbaas-mongodb.database.restore Restore databases by SRE as delegated by service instance owner 
 hyperp-dbaas-mongodb.backups.cos.enable Enable backups to IBM Cloud Object Storage 
 hyperp-dbaas-mongodb.backups.cos.disable Disable Backups to IBM Cloud Object Storage 
 hyperp-dbaas-mongodb.backups.restore Restore databases from backup 
 hyperp-dbaas-mongodb.database.backup.cos Back up databases to IBM Cloud Object Storage 



For the events of hyperp-dbaas-mongodb.cluster.create and hyperp-dbaas-mongodb.cluster.delete, the Activity Tracker service doesn't record the account name and IP address of the user who performs the action. The values of initiator.name and host.address in the log indicate the service ID of Cloud Broker and the IP address of Cloud Broker.
Our example does this with 10 million keys in our database, which doesn't take much time to migrate depending on your bandwidth.





 Getting the destination and source database credentials 

Now, you need to have the credentials of both the old source database and your new Databases for Redis deployment. You can get the credentials for both Databases for Redis deployments by clicking the respective databases from the IBM Cloud resources page. Then, click the Service credentials link from the left menu that takes you to the Service credentials view. From there, you can create a new credential for the destination database by clicking the New credential button. You can use any credentials that you’ve already created for the source respectively.

Zoom

![Service credentials view](https://cloud.ibm.com/docs-content/v1/content/c1b02d6157702384a9668d8c58bb951e69117793/databases-for-redis/images/service-credentials.png)

Figure 1. Service credentials view

Another way to get this information is using the IBM Cloud CLI. Using the cdb plug-in, run:

ibmcloud cdb deployment-connections <Redis deployment name>

This provides you with your Databases for Redis connection URI that includes the hostname and port. To get the decoded CA certificate for the database, run:

ibmcloud cdb deployment-cacert <Redis deployment name>

After the CA certificate is decoded, you need to save that to a file to connect to the database later. If you don’t know the password for your deployment, either get that from your generated service credentials or you can create a new password by running:

ibmcloud cdb deployment-user-password <Redis deployment name> admin <new password>

With this information, you have what you need for the destination and the source databases.





 Running the script and migrating data 

Since you have all the credentials for both databases (source and new Databases for Redis destination), you're now ready to run the script. The Python script file name is pymigration.py.