vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* UI
* CLI







 Dedicated host profiles 

A dedicated host profile is a combination of attributes, such as the number of vCPUs, amount of RAM, and optionally instance storage that define the amount of compute capacity that is provided in a host that is provisioned exclusively for your use.

Dedicated hosts are available in various profile sizes to best suit the size and scale of your workloads: Balanced, Compute, Memory, Very High Memory, and Ultra High Memory. All virtual server instances that are provisioned on the dedicated host are provisioned from the same family and class of profiles. For example, if you choose a Memory profile for your dedicated host, all instances that are provisioned on the host must also be created with a Memory virtual server profile. If you choose a Compute with instance storage dedicated host, all instances that are provisioned on the host must also be created with Compute with instance storage virtual server profiles. This set up ensures you can always use the entire dedicated host resource.

The Madrid region only supports dedicated host profiles with instance storage.

Dedicated hosts are not supported for LinuxONE (s390x processor architecture).

The following profile families are available:



Table 1. Dedicated host family selections

 Families Description 

 [Balanced](https://cloud.ibm.com/docs/vpc?topic=vpc-dh-profilesbalanced-dh-pr) Best for midsize databases and common cloud applications with moderate traffic. Balanced profiles offer vCPU to memory ratios with 1 vCPU to 4 GiB of RAM. Balanced profiles are offered with and without instance storage. 
 [Compute](https://cloud.ibm.com/docs/vpc?topic=vpc-dh-profilescompute-dh-pr) Best for workloads with intensive CPU demands, such as medium to high web traffic workloads, production batch processing, and front-end web servers. Compute profiles offer vCPU to memory ratios with 1 vCPU to 2 GiB of RAM. Compute profiles are offered with and without instance storage.
Connecting with psql 

You can access your IBM Cloud® Databases for EnterpriseDB database directly from its command line client, psql. You can use psql for direct interaction and monitoring of the data structures that are created within the database. It is also useful for testing and monitoring the queries and performance, installing and modifying scripts, and other management activities.

The admin user comes with the Databases for EnterpriseDB default role [pg_monitor](https://www.postgresql.org/docs/10/static/default-roles.html), allowing access to Databases for EnterpriseDB monitoring views and functions. By default, the admin user does not have permissions on objects that are created by other users.

You must set the admin password before you use it to connect to the database. For more information, see the [Setting the Admin Password](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-admin-password) page.



 Installing psql 

Install the command line client for Databases for EnterpriseDB, psql. To use psql, the Databases for EnterpriseDB client tools need to be installed on the local system. They can be installed with the full PostgreSQL package that is provided from [postgresql.org](https://www.postgresql.org/download/), or as a [package from your operating system's package manager](https://www.compose.com/articles/postgresql-tips-installing-the-postgresql-client/).

For more information about psql, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/static/app-psql.html).





 psql Connection Strings
* UI
* CLI







 Dedicated host profiles 

A dedicated host profile is a combination of attributes, such as the number of vCPUs, amount of RAM, and optionally instance storage that define the amount of compute capacity that is provided in a host that is provisioned exclusively for your use.

Dedicated hosts are available in various profile sizes to best suit the size and scale of your workloads: Balanced, Compute, Memory, Very High Memory, and Ultra High Memory. All virtual server instances that are provisioned on the dedicated host are provisioned from the same family and class of profiles. For example, if you choose a Memory profile for your dedicated host, all instances that are provisioned on the host must also be created with a Memory virtual server profile. If you choose a Compute with instance storage dedicated host, all instances that are provisioned on the host must also be created with Compute with instance storage virtual server profiles. This set up ensures you can always use the entire dedicated host resource.

The Madrid region only supports dedicated host profiles with instance storage.

Dedicated hosts are not supported for LinuxONE (s390x processor architecture).

The following profile families are available:



Table 1. Dedicated host family selections

 Families Description 

 [Balanced](https://cloud.ibm.com/docs/vpc?topic=vpc-dh-profiles&interface=uibalanced-dh-pr) Best for midsize databases and common cloud applications with moderate traffic. Balanced profiles offer vCPU to memory ratios with 1 vCPU to 4 GiB of RAM. Balanced profiles are offered with and without instance storage. 
 [Compute](https://cloud.ibm.com/docs/vpc?topic=vpc-dh-profiles&interface=uicompute-dh-pr) Best for workloads with intensive CPU demands, such as medium to high web traffic workloads, production batch processing, and front-end web servers. Compute profiles offer vCPU to memory ratios with 1 vCPU to 2 GiB of RAM. Compute profiles are offered with and without instance storage.
If custom metadata was associated with the document when it was uploaded to v1, it is also present in the JSON file. In addition, the file contains several fields that were generated by the v1 analysis. Retain only a subset of this data as part of the document that you add to Discovery v2.

The following tips can help you decide which fields to keep:



* Include the text field or any other field with textual content that you want to be able to enrich or search in Discovery v2.
* Include any custom metadata that is stored in the document. This metatdata is typically specific to the application that uses Discovery and is used to filter documents in a search. For example, metadata.customer_id.
* Do not include enrichments from Discovery v1. For example, enriched_text.entities. Discovery v2 generates its own enrichments.
* Exclude fields that are generated by Discovery unless they are used by your application and contain information that is unique to the v1 version of the document. In that case, rename the field so that it does not get replaced when the document is ingested into Discovery v2. For example, extracted_metadata.publicationdate is a field that is generated by Discovery when a document is ingested. Maybe you want to retain the metadata.parent_document_id information from v1 to understand how subdocuments were originally generated from a single source document.
* Avoid fields that have reserved field names. For more information, see [How fields are handled](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfield-name-limits).



4. Ingest each edited v1 JSON document into the Discovery v2 instance. The Discovery v1 document ID can be maintained in Discovery v2. For more information about how to retain the document ID, see [Retaining document IDs](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-migrate-to-v2migrate-to-v2-keep-docid).









 Transferring relevancy training 

Relevancy training that was done in Discovery v1 can be transferred to Discovery v2.
Type acurl $URL/_all_dbs to see an array of databases.



A quick note here on formatting JSON on the command line. We can send the output of our acurl command to another tool, which formats the data nicely on the terminal. The following tools are available for your use:



* Jq available from the URL on the page, which is more than just a JSON formatter - it allows JSON to be parsed, queried, and manipulated too.
* python -m json.tool is a simple JSON formatter, if Python is installed on your computer.



So acurl $URL/_all_dbs | jq means pipe the output of acurl into jq and what you see is a nicely formatted, colored output.

The IBM Cloudant API paths are hierarchical with the first level that gives you information about the service, and then each database sits at a level beneath it.

So acurl $URL/books gives us information about the books database that we created earlier.

You see information about how many documents it has, how many deleted documents, and how much disk space it's occupying.

Don't forget to pipe the output to jq or Python to get a prettier output.

If we want to see the documents contained in the database, we can use the _all_docs endpoint.

So acurl $URL/books/_all_docs means get all the documents from the books database from the IBM Cloudant service at the supplied URL.

This command's results return a list of _id and _rev values for each document. If you want the document bodies too, then add ?include_docs=true to your API call.

If we want to fetch a single document back from the database, then documents sit one level beneath the database in the hierarchy of the URL.

So acurl $URL/books/id means "get document ID from the database books from the IBM Cloudant service at the supplied URL".

Notice the hierarchy: service, database, and document.

So far we only used the GET HTTP method, which is the default one for curl and the one used when you enter a URL into your web browser.
ibmcloud cbr zone-update <ZONE-ID> --addresses=1.2.3.4 --name=<NAME>

Updating requires the ZONE-ID, not the zone name. Use the following command to list your zones and retrieve the relevant ZONE-ID:

ibmcloud cbr zones

The zone-update command is an overwrite. Include all of the fields that are required as if you are creating the rule from scratch. If you omit any required fields, the rule overwrites those missing fields as empty, and the rule might fail because some of those fields are required, regardless of whether they are changing the rule.

Delete a zone by using a command like:

ibmcloud cbr zone-delete <ZONE-ID>







 Creating rules 

Rules restrict access to specific cloud resources based on resource attributes and contexts. A created rule can accept up to 2,000 IP/CIDR values for private endpoints and up to 2,000 IP/CIDR values for public endpoints. This limit is specfic to Cloud Databases. Other IBM Cloud® service limits may vary.

Cloud Databases does not support IPv6 addresses. If an IPv6 address is included, it will be ignored.



 Creating rules in the UI 



1. Go to Manage > Context-based restrictions in the IBM Cloud® console.
2. Select Rules.
3. Click Create.
4. Select Specific APIs and then select Data plane. Any other selection results in an error.

Cloud Databases does not currently support Control plane as an option.
5. Click Next.
6. Scope the rule to All resources or Specific resources. For more information, see [Protecting Cloud Databases resources](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-cbrcbr-overview-protect-services).
7. Click Continue.
8. Define the allowed endpoint types.



* Keep the toggle set to No to allow all endpoint types.
* Set the toggle to Yes to allow only specific endpoint types, then choose from the list.



9.
For example, if your service cluster has both a Databases for etcd and a Databases for PostgreSQL instance on the same service cluster, the name is based on your Databases for etcd instance.

ICD enabled by IBM Cloud Satellite supports Satellite locations on [Amazon Web Services (AWS)](https://cloud.ibm.com/docs/satellite?topic=satellite-aws) and [on-premises](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-satellite-on-prem).



 An Overview 

Zoom

![An Overview of ICD enabled by IBM Cloud Satellite](https://cloud.ibm.com/docs-content/v1/content/c1b02d6157702384a9668d8c58bb951e69117793/databases-for-redis/includes/cloud-databases/includes/cloud-databases/images/sat_on_icd.svg)

Image 1. An Overview of ICD enabled by IBM Cloud Satellite





 Before you begin 

ICD enabled by IBM Cloud Satellite does not yet provide the Security and Compliance integration, Activity Tracking, or Metrics Monitoring of a standard IBM Cloud® Databases deployment.



* Refer to the [Satellite Usage requirements](https://cloud.ibm.com/docs/satellite?topic=satellite-requirements).
* Be sure you set up the [IBM Cloud CLI](https://cloud.ibm.com/docs/satellite?topic=satellite-setup-cli), the plug-in for Satellite commands, and other related CLIs.
* If you did not create a Satellite location, see [Setting up Satellite locations](https://cloud.ibm.com/docs/satellite?topic=satellite-locations). Follow the steps in the [Manually creating Satellite locations](https://cloud.ibm.com/docs/satellite?topic=satellite-locationslocation-create) documentation.



* For the management location, choose Washington DC.
For example, if your service cluster has both a Databases for etcd and a Databases for PostgreSQL instance on the same service cluster, the name is based on your Databases for etcd instance.

ICD enabled by IBM Cloud Satellite supports Satellite locations on [Amazon Web Services (AWS)](https://cloud.ibm.com/docs/satellite?topic=satellite-aws) and [on-premises](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-satellite-on-prem).



 An Overview 

Zoom

![An Overview of ICD enabled by IBM Cloud Satellite](https://cloud.ibm.com/docs-content/v1/content/3505da55c475bf1edbe3eb3496324ba13941b783/databases-for-postgresql/includes/cloud-databases/includes/cloud-databases/images/sat_on_icd.svg)

Image 1. An Overview of ICD enabled by IBM Cloud Satellite





 Before you begin 

ICD enabled by IBM Cloud Satellite does not yet provide the Security and Compliance integration, Activity Tracking, or Metrics Monitoring of a standard IBM Cloud® Databases deployment.



* Refer to the [Satellite Usage requirements](https://cloud.ibm.com/docs/satellite?topic=satellite-requirements).
* Be sure you set up the [IBM Cloud CLI](https://cloud.ibm.com/docs/satellite?topic=satellite-setup-cli), the plug-in for Satellite commands, and other related CLIs.
* If you did not create a Satellite location, see [Setting up Satellite locations](https://cloud.ibm.com/docs/satellite?topic=satellite-locations). Follow the steps in the [Manually creating Satellite locations](https://cloud.ibm.com/docs/satellite?topic=satellite-locationslocation-create) documentation.



* For the management location, choose Washington DC.
Creating Views (MapReduce) 

Views are used to obtain data stored within a database. Within views, you use reduce functions, map and reduce functions, and storing a view definition.

Learn more about the simplest view, reduce functions, map and reduce function restrictions, and storing a view definition. Plus, see the examples that are provided. Views are written by using JavaScript.



 View concepts 

Views are mechanisms for working with document content in databases. A view can selectively filter documents and speed up the search for content. It can be used to pre-process the results before they're returned to the client.

Views are simply JavaScript functions, which are defined within the views field of a design document. When you use a view, or more accurately when you run a query by using your view, the system applies the JavaScript function to every document in the database. Views can be complex. You might choose to define a collection of JavaScript functions to create the overall view that is required.





 View index partitioning type 

A view index inherits the partitioning type from the options.partitioned field of the design document that contains it.





 A simple view 

The simplest form of view is a map function. The map function produces output data that represents an analysis (a mapping) of the documents that are stored within the database.

For example, you might want to find out which user completed the online registration and has a verified email to contact. You might find this information by inspecting each document, and looking for a field in the document called "email_verified" and getting the value of "email". If the field is present and has the value true, it means the user completed the registration, and you can contact them by email. If the field isn't present or has a value of something else than true, the user didn't complete the registration.

Using the emit function in a view function makes it easy to produce a list in response to running a query by using the view. The list consists of key and value pairs, where the key helps you identify the specific document and the value provides just the precise detail you want. The list also includes metadata such as the number of key:value pairs returned.
If you don't check for conflicts, or don't fix them, your IBM Cloudant database has the following problems:



* Document inconsistency increases because conflicting documents continue to multiply.
* Database size increases because documents with conflicts must be kept until the conflict is resolved.
* Performance degrades because it takes more work for IBM Cloudant to respond to each request since it has to go through all the conflicted documents to find the "best possible" version.







After you find a conflict, follow these four steps to resolve it.



1. [Get](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-document-versioning-and-mvccget-conflicting-revisions-mvcc) the conflicting revisions.
2. [Merge](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-document-versioning-and-mvccmerge-the-changes-mvcc) them into your application or ask the owner what to do.
3. [Upload](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-document-versioning-and-mvccupload-the-new-revision-mvcc) the new revision.
4. [Delete](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-document-versioning-and-mvccdelete-old-revisions-mvcc) old revisions.





* What is eventual consistency?

IBM Cloudant databases are [eventually consistent](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-cap-theoremcap-theorem), which means IBM Cloudant must ensure that no differences exist between nodes. These inconsistencies can happen when out-of-date documents are synchronized.

It's important for IBM Cloudant databases to have concurrent read and write access. MVCC enables that capability.
* [Version 0.1.585](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-log-01585)
* [Version 1.0.2](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-logv1-102)
* [Version 0.1.584](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-log-01584)
* [Version 1.0.1](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-logv1-101)
* [Version 0.1.583](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-log-01583)
* [Version 1.0.0](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-logv1-100)
* [Version 0.1.582](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_cli_change_logcli-change-log-01582)







 High availability and disaster recovery 

[High availability](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-dr)



* [Ownership of responsibilities](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-responsibilities)
* [What level of availability do I need?](https://cloud.ibm.com/docs/Registry?topic=Registry-ha-drha-level)
* [What level of availability does IBM Cloud offer?]
The filesystems used by Linux, UNIX, and UNIX-like computer systems can contain special types of files, such as block and character device nodes and files that represent named pipes, which cannot be crawled because they do not contain data, but serve as device or I/O access points. Attempting to crawl such files generate errors during the crawl. To avoid such errors, exclude the /dev directory in any top-level crawl on a Linux, UNIX, or UNIX-like filesystem. If present on the system that you are crawling, also exclude temporary system directories, such as /proc, /sys, and /tmp, that contain transient files and system information. hops - Internal use only. default-allow - Internal use only.









 Configuring Database crawl options 

The database connector allows you to crawl a database by executing a custom SQL command and creating one document per row (record) and one content element per column (field). You can specify a column to be used as a unique key, as well as a column containing a timestamp representing the last-modification date of each record. The connector retrieves all records from the specified database, and can also be restricted to specific tables, joins, and so on in the SQL statement.

The Database connector allows you to crawl the following databases:



* IBM® DB2
* MySQL
* Oracle PostgreSQL
* Microsoft SQL Server
* Sybase
* Other SQL-compliant databases, via a JDBC 3.0-compliant driver



The connector retrieves all records from the specified database and table.

JDBC Drivers - The Database connector ships with Oracle JDBC (Java Database Connectivity) driver version 1.5. All third-party JDBC drivers shipped with the Data Crawler are located in the connectorFramework/crawler-connector-framework-../lib/java/database directory of your Data Crawler installation, where you can add, remove, and modify them as necessary. You can also use the extra_jars_dir setting in the crawler.conf file to specify another location.

DB2 JDBC Drivers - The Data Crawler does not ship with the JDBC drivers for DB2 due to licensing issues.
If an unhealthy IP address is detected, traffic is not routed to this IP address anymore.







 What if I want to load balance workloads on the private network? 

IBM Cloud does not offer a global load balancer service on the private network. However, you can connect your cluster to a private load balancer that you host in your on-prem network by using one of the [supported VPN options](https://cloud.ibm.com/docs/containers?topic=containers-vpn). Make sure to expose your apps on the private network by using [Application Load Balancers (ALBs)](https://cloud.ibm.com/docs/containers?topic=containers-managed-ingress-aboutmanaged-ingress-albs) or [Network Load Balancers (NLBs)](https://cloud.ibm.com/docs/containers?topic=containers-loadbalancer-about), and use the private IP address in your VPN settings to connect your app to your on-prem network.