vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Auditors reviewed Databases for etcd for compliance under PCI DSS version 3.2.1 at Service Provider Level 1.

Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using Databases for etcd. Customers can request and use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications. It is the responsibility of the customer to document and operate CDEs and applications that are built by using IBM Cloud Platform services in a PCI DSS-compliant manner.

It is the customer’s responsibility to familiarize themselves with these processes and to manage data retention and removal from the service according to the customer’s policies.

A full list of PCI DSS-ready IBM Cloud Platform services, and options to request a PCI DSS AOC and SRM guide, can be found at the [IBM Cloud compliance page](https://www.ibm.com/cloud/compliance/industry).





 Terms 



* [The IBM Privacy Policy](https://www.ibm.com/privacy/us/en/)
* [The IBM Cloud Notices and Terms of Use](https://cloud.ibm.com/docs/overview/terms-of-use?topic=overview-terms)
Auditors reviewed Databases for PostgreSQL for compliance under PCI DSS version 3.2.1 at Service Provider Level 1.

Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using Databases for PostgreSQL. Customers can request and use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications. It is the responsibility of the customer to document and operate CDEs and applications that are built by using IBM Cloud Platform services in a PCI DSS-compliant manner.

It is the customer’s responsibility to familiarize themselves with these processes and to manage data retention and removal from the service according to the customer’s policies.

A full list of PCI DSS-ready IBM Cloud Platform services, and options to request a PCI DSS AOC and SRM guide, can be found at the [IBM Cloud compliance page](https://www.ibm.com/cloud/compliance/industry).





 Terms 



* [The IBM Privacy Policy](https://www.ibm.com/privacy/us/en/)
* [The IBM Cloud Notices and Terms of Use](https://cloud.ibm.com/docs/overview/terms-of-use?topic=overview-terms)
Auditors reviewed Databases for EnterpriseDB for compliance under PCI DSS version 3.2.1 at Service Provider Level 1.

Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using Databases for EnterpriseDB. Customers can request and use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications. It is the responsibility of the customer to document and operate CDEs and applications that are built by using IBM Cloud Platform services in a PCI DSS-compliant manner.

It is the customer’s responsibility to familiarize themselves with these processes and to manage data retention and removal from the service according to the customer’s policies.

A full list of PCI DSS-ready IBM Cloud Platform services, and options to request a PCI DSS AOC and SRM guide, can be found at the [IBM Cloud compliance page](https://www.ibm.com/cloud/compliance/industry).





 Terms 



* [The IBM Privacy Policy](https://www.ibm.com/privacy/us/en/)
* [The IBM Cloud Notices and Terms of Use](https://cloud.ibm.com/docs/overview/terms-of-use?topic=overview-terms)
Or you can use the [Resource Controller API](https://cloud.ibm.com/apidocs/resource-controller), with a PATCH request to the [/resource_instances/{id}](https://cloud.ibm.com/apidocs/resource-controllerupdate-a-resource-instance) endpoint.

Changing the type of endpoints available on your deployment does not cause any downtime from a database perspective. However, if you disable an endpoint that is being used by you or your applications, those connections are dropped.





 Credentials for Private Endpoints 

You can use either public or private connection strings with any set of credentials you make on your deployment. By default, the connection strings for a set of credentials are filled with strings for connecting over a public endpoint. If you are using private endpoints, you can specify connection strings that contain the private endpoint be generated instead.

When you create credentials in Service Credentials, use either the { "service-endpoints": "public" } or the { "service-endpoints": "private" } parameter to specify which endpoint gets filled into the connection strings.

In the API, you can use the [/deployments/{id}/users/{userid}/connections/{endpoint_type}](https://cloud.ibm.com/apidocs/cloud-databases-apidiscover-connection-information-for-a-deployment-f-e81026) to retrieve connection strings for both public or private endpoints.

If you only have private endpoints on your deployments, then all new credentials have private endpoints in the connection strings.





 Connecting Through Private Endpoints 

IBM Cloud® Databases offer both private and public cloud service endpoints. If you want to run your application or access the end point from a browser that is not on the private network, you must take these additional steps:



* Ensure your Cloud IaaS or SL account is [enabled for private endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview).
* Create a virtual machine (VSI) that runs Linux
* Configure a user account with SSH access
The potential for rewinds makes the changes feed unsuitable for an application that expects queue-like behavior.

To reiterate, IBM Cloudant's changes feed promises to deliver a document at least one time in a changes feed, and gives no guarantees about repeated values across multiple requests.





The changes feed doesn't guarantee how quickly an incoming change appears to a client that consumes the changes feed. Applications must not be developed with the assumption that data inserts, updates, and deletes are immediately propagated to a changes reader.





If a document is updated several times in between changes feed calls, then the changes feed might reflect only the most recent of these changes. The client does not receive every change to every document.

The IBM Cloudant changes feed isn't a transaction log that contains every event that happened in time order.





Filtering the changes feed, and by extension, running filtered replication has its uses:



* Copying data from source to target but ignoring deleted documents.
* Copying data but without index definitions (design documents).



This [blog post](https://blog.cloudant.com/2019/12/13/Filtered-Replication.html) describes how supplying a selector during replication makes work of these use cases run smoothly.

The changes feed with an accompanying selector parameter is not the way to extract slices of data from the database on a routine basis. It must not be used as a means of running operational queries against a database. Filtered changes are slow (the filter is applied to every changed document in turn, without the help of an index). This process is much slower than creating a secondary index (such as a MapReduce view) and querying that view.





No, IBM Cloudant does not guarantee connection duration for a continuous changes feed. It might be regularly disconnected by the server for any number of reasons, which include maintenance, security, or network errors. Code that uses the changes feed must be designed to use a recently saved sequence ID as a since value to make a new request to resume the changes feed after an error or disconnection.





If the use case is based on the following statement, then this result cannot be achieved with the IBM Cloudant changes feed.

"Fetch me every document that has changed since a known date, in the order they were written."
How do I stop relevancy training from being applied to my project? 

Use the API to delete the relevancy model that is associated with your project. To delete the model, you delete that training data that is associated with the ranker model. For more information, see [Deleting training queries](https://cloud.ibm.com/apidocs/discovery-datadeletetrainingqueries).





 Does relevancy training impact passage search? 

No. Relevancy training is used for document search only. It has no impact on passage search.





 Does relevancy training impact answer finding? 

Not directly. Relevancy training indirectly impacts answer finding because it changes the order of the documents from which answers are retrieved. It reranks the returned documents from most to least relevant.





 How do I check errors and warnings? 

Open the Manage collections page. Choose your collection, then open the Activity tab.





 How do I interpret the confidence score that appears in natural language query results after training? 

See [confidence scores](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-trainconfidence).







 Interpreting relevancy training errors and warnings 

The following list has explanations for some common error and warning messages.



 Warning: Invalid training data found: The document was not returned in the top 100 search results for the given query, and will not be used for training 

This warning occurs when the document_ids in your training data do not match the document_ids in a search that is performed against the collection. Check your queries to make sure that the document_id of the document you are rating is returned in the top 100 results for that query. If it is not, then you might want to check two things:



* If the document is not returned in the top 100, it might not be an example of a high-quality result. Reevaluate whether to use the document.
* If the document is not returned at all, then review why it is not returned and see whether any text in the document matches portions of the query.



This warning indicates that you might have one or more failed queries.
* Within each data center, IBM Cloudant already offers high availability by storing data in triplicate across three servers.
* Replication occurs at the database rather than account level and must be explicitly configured.
* IBM Cloudant doesn't provide any Service Level Agreements (SLAs) or certainties about replication latency. IBM Cloudant doesn't monitor individual replications. Your own strategy for detecting failed replications and restarting them is advisable.





 Before you begin an active-active deployment 

For an active-active deployment, a strategy for managing conflicts must be in place, so be sure to understand how [replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-apireplication-api) and [conflicts](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-document-versioning-and-mvccdocument-versioning-and-mvcc) work before you consider this architecture.

Go to the [IBM Cloud Support portal](https://www.ibm.com/cloud/support) if you need help with how to model data to handle conflicts effectively.





 Overview 

In the following material, a bidirectional replication is created. This configuration allows two databases to work in an active-active topology.

The configuration assumes that you have two accounts in different regions:



* myaccount-dc1.cloudant.com
* myaccount-dc2.cloudant.com



After these accounts are created, follow these steps:



1. [Create](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-configuring-ibm-cloudant-for-cross-region-disaster-recoverystep-1-create-your-databases) a pair of peer databases within the accounts.
2. [Set up](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-configuring-ibm-cloudant-for-cross-region-disaster-recoverystep-2-create-an-api-key-for-your-replications) API keys to use for the replications between these databases.
3.
Service Endpoints Integration 

All Cloud Databases deployments offer integration with [IBM Cloud Service Endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview). It gives you the ability to enable connections to your deployments from the public internet and over the IBM Cloud Private network.

Service Endpoints are available in all IBM Cloud Multi-Zone Regions and some single-campus multizone regions. Deployments in all other regions are able to use Service Endpoints.



 Public Endpoints 

Public endpoints provide a connection to your deployment on the public network. At provision time, a public endpoint is the default option for all deployments. Your environment needs to have internet access to connect to a deployment.





 Private Endpoints 

A deployment with a service endpoint on the private network gets an endpoint that is not accessible from the public internet. All traffic is routed to hardware dedicated to Cloud Databases deployments and remains on the IBM Cloud Private network. All traffic to and from this endpoint is free and unmetered on the condition that the traffic remains in IBM Cloud. After your environment has access to the IBM Cloud Private network, an internet connection is not required to connect to your deployment.

Database instances with private endpoints are reachable from any account within the private network and access to each instance requires authentication. To restrict this access to specific IP addresses, or ranges of IP addresses, configure [allowlisting](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-allowlisting).

A MongoDB deployment cannot support both [public and private endpoints simultaneously](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-service-endpoints&interface=uiprovisioning-service-endpoints). This cannot be changed after provisioning.

For more information on accessing services using service endpoints, see [Secure access to services using service endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview).





 Enabling Service Endpoints
Each shard holds its documents in a large [B-tree](https://en.wikipedia.org/wiki/B-tree) on disk. Indexes are stored in the same way. As more documents are added to a shard, the number of steps that are used to traverse the B-tree during a typical document lookup or query increases. This 'depth increase' tends to slow down requests because more data must be read from caches or disk.

In general, avoid having more than 10 million documents per shard. In terms of overall shard size, keeping shards under 10 GB is helpful for operational reasons. For example, smaller shards are easier to move over the network during rebalancing.

Given the conflicting requirements to avoid having too many documents and keeping shard size low, a single Q value cannot work optimally for all cases. IBM Cloudant tunes the defaults for clusters over time as usage patterns change.

Nevertheless, for a specific database, it is often useful to take the time to consider observed request patterns and sizing. You can use this information to guide the future selection of an appropriate number of shards. Testing with representative data and request patterns is essential for better estimation of good Q values. Be prepared for production experience to alter those expectations.

The following simple guidelines might be helpful during the early planning stages. Remember to validate your proposed configuration by testing with representative data, particularly for larger databases:



* If your data is trivial in size, such as a few tens or hundreds of MB, or thousands of documents, then there is little need for more than a single shard.
* For databases of a few GB or few million documents, then a single-digit shard count such as 8 is likely to be acceptable.
* For larger databases of tens to hundreds of millions of documents or tens of GB, consider configuring your database to use 16 shards.
* For even larger databases, consider manually sharding your data into several databases. For such large databases, go to the [IBM Cloud Support portal](https://www.ibm.com/cloud/support) for advice.



The numbers in these guidelines are derived from observation and experience rather than precise calculation.





 Working with shards 



 Setting shard count
[High availability](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-dr)



* [Ownership of responsibilities](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-responsibilities)
* [What level of availability do I need?](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-level)
* [What level of availability does IBM Cloud offer?](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-service)
* [Locations for service availability](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-locations)
* [Frequently asked questions about high availability](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-dr_faq)



* [Does IBM Cloud replicate the service?](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-dr_service)
* [Are users required to replicate the service?](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-dr_service_replicate)





[Business continuity and disaster recovery](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr)



* [Your responsibilities when you're using Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr-responsibilities)
* [Disaster recovery strategy](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr-strategy)
* [Locations for service availability](https://cloud.ibm.com/docs/Registry?topic=Registry-bc-drbc-dr-locations)
Checkpoints for replication are [enabled by default](https://cloud.ibm.com/apidocs/cloudantpostreplicate), which means that replicating the source database requires write access.

To enable authentication during replication, include a username and password in the database URL. The replication process uses the supplied values for HTTP Basic Authentication.

See the following example of specifying username and password values for accessing source and target databases during replication:

{
"source": "https://$USERNAME:$PASSWORD@example.com/db",
"target": "https://$USERNAME:$PASSWORD@$ACCOUNT.cloudant.com/db"
}





 Filtered replication 

Sometimes you don't want to transfer all documents from source to target. To choose which documents to transfer, include one or more filter functions in a design document on the source. You can then tell the replicator to use these filter functions.

Filtering documents during replication is similar to the process of [filtering the _changes feed](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-design-documentsfilter-functions).

A filter function takes two arguments:



* The document to be replicated.
* The replication request.



A filter function returns a true or false value. If the result is true, the document is replicated.

To set up filtering, use the selector field whenever possible. When you use the selector field, you can specify a filter without having to replicate the entire database. This method makes filtering faster and causes less load on IBM Cloudant. For more information, see the [selector field](https://cloud.ibm.com/apidocs/cloudantpostfind) documentation.

See the following example of a filter function:

function(doc, req) {
return !!(doc.type && doc.type == "foo");
}

Filters are stored under the topmost filters key of the design document.

See the following example of storing a filter function in a design document:

{
"_id": "_design/myddoc",
Deleting a document 

The steps shown here demonstrate how to delete a document.



1. Send a DELETE request with the document's most recent _rev in the query string.
2. Run the following command: https://$ACCOUNT.cloudant.com/$DATABASE/$DOCUMENT_ID. The response contains the ID and the new revision of the document, or an error message if the delete failed.



If you fail to provide the most recent _rev, IBM Cloudant responds with a [409 error](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes). This error prevents you overwriting data that were changed by other clients. If the write [quorum](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-delete-a-docquorum-writing-and-reading-data) can't be met, a [202 response](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes) is returned.

IBM Cloudant doesn't completely delete the specified document. Instead, it leaves a [tombstone](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-delete-a-doctombstone-documents) with basic information about the document. The tombstone is required so that the delete action can be replicated to other copies of the database. Since the tombstones stay in the database indefinitely, creating new documents and deleting them increases the disk space usage of a database. They might also increase the query time for the primary index, which is used to look up documents by their ID.

The following steps show you how to delete a request by using HTTP.



1. Send a DELETE request with the document's most recent _rev in the query string by using HTTP.
2. Run the following command: DELETE /$DATABASE/$DOCUMENT_ID?rev=$REV HTTP/1.1.



The following code examples show you how to delete a document by using the command line.



* Curl
* Java
* Python
* Go
You can create up to 20 API keys.

For security reasons, the API key is only available to be copied or downloaded at the time of creation. If the API key is lost, you must create a new API key.





 Step 2: Clone the project 

Clone the project from the Cloud Databases [Hello World project GitHub repository](https://github.com/IBM-Cloud/clouddatabases-helloworld-examples).

git clone https://github.com/IBM-Cloud/clouddatabases-helloworld-examples.git





 Step 3: Install the infrastructure 

In this step, you deploy an instance of the database service you want to use. The GitHub repository contains folders for various Cloud Databases resources.



1. From the main GitHub project folder, navigate into the terraform service folder of your choice, for example, mysql/terraform.
2. On your machine, create a document that is named terraform.tfvars, with the following fields:

ibmcloud_api_key = "<your_api_key_from_step_1>"
region = "<your_region>"
admin_password = "<make_up_a_password>"

The terraform.tfvars document contains variables that you might want to keep secret so it is ignored by the GitHub repository.
3. Install the infrastructure with the following command:

terraform init
terraform apply --auto-approve

The Terraform script outputs configuration data that is needed to run the application, so copy it into the root folder:

terraform output -json >../config.json







 Step 4: Run your app locally 



1. To connect to the database from your local machine, ensure that you are in your service folder, then install the node dependencies and run the service with the following commands:

npm install

npm run start

If successful, the output shows you are connected:

Connected!
Server is listening on port 8080
2.