vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Dependencies 



 Dependencies to other IBM Cloud Services 

In general, customer data is not leaving the region or country, in which an instance of IBM Cloud® Data Engine was provisioned. This rule excludes customer data that is used by dependant service instances that are under customer responsibility.



 Critical Dependencies 

The following dependencies of Data Engine are considered critical. Any loss of connectivity or service of one of these dependencies results in a functional impact to the customer on Data Engine.



Table 1. Critical

 Service name Description 

 Business Support Services for IBM Cloud (BSS) Used to access information about the IBM Cloud® account, service subscription, service usage, and billing. 
 IBM® Analytics Engine (IAE) Spark engine used to run the SQL queries. To learn more, see [IBM Analytics Engine](https://cloud.ibm.com/docs/AnalyticsEngine). 
 IBM® Cloudant® for IBM Cloud® Used to store Data Engine's job status, storing SQL query texts, execution, and error information. To learn more, see [IBM Cloudant](https://cloud.ibm.com/docs/Cloudant). 
 IBM Cloud® Databases for PostgreSQL Used for catalog store (Hive Metastore). Stores metadata that describes customer IBM Cloud® Object Storage data as Spark tables and views. Lo learn more, see [IBM Cloud Databases for PostgreSQL](https://cloud.ibm.com/docs/databases-for-postgresql). 
 IBM Cloud® Identity and Access Management (IAM) To authenticate requests to the service and authorize user actions, Data Engine implements platform and service access roles in Identity and Access Management. To learn more, see [Cloud Identity and Access Management](https://cloud.ibm.com/docs/account?topic=account-iamoverview). 
 IBM Cloud® Internet Services (CIS) Used to manage our external network connectivity, our ibm.com domains, and distributed proxies with TLS termination to support DDoS, WAF, and Caching for HTTPs endpoints. To learn more, see [IBM Cloud Internet Services](https://cloud.ibm.com/docs/cis).
_balancers](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/cis_global_load_balancers)<br> * [ibm_cis_custom_pages](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/cis_custom_pages)<br> * [ibm_dns_glbs](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/dns_glb)<br> * [ibm_dns_glb_monitors](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/dns_glb_monitors)<br> * [ibm_dns_glb_pools](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/dns_glb_pools)<br> * [ibm_is_vpc_default_routing_table](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_vpc_default_routing_table)<br> * [ibm_is_vpc_routing_tables](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_vpc_routing_tables)<br> *
With IBM Watson® Natural Language Understanding, developers can analyze semantic features of text input, including categories, concepts, emotion, entities, keywords, metadata, relations, semantic roles, and sentiment.



 Provision Object Storage and Natural Language Understanding services 



1. Create an instance of [Object Storage](https://cloud.ibm.com/catalog/services/cloud-object-storage)



1. Select the Lite plan or the Standard plan if you already have an Object Storage service instance in your account.
2. Set Service name to code-engine-cos.
3. Select the resource group where you created the Code Engine project.
4. Click on Create.
5. Capture the service name in a shell variable:

COS_INSTANCE_NAME=YourServiceName



2. Click Buckets then Customize your bucket

When you create buckets or add objects, be sure to avoid the use of Personally Identifiable Information (PII).Note: PII is information that can identify any user (natural person) by name, location, or any other means.



1. Enter Unique bucket name such as <your-initials>-bucket-code-engine.
2. Select a Location, where you created the Code Engine project.
3. Select Smart Tier Storage class.
4. Click Create bucket.
5. Capture the bucket name in a shell variable:

COS_BUCKETNAME=your-initials-bucket-code-engine



3. Click to open the bucket.



1. Click the Configuration tab
2. The Direct endpoint will keep data within the IBM cloud. Capture the direct endpoint in a shell variable. In the Dallas, us-south, region it might be:

COS_ENDPOINT=s3.direct.us-south.cloud-object-storage.appdomain.cloud



4. Create an instance of [Natural Language Understanding](https://cloud.ibm.com/catalog/services/natural-language-understanding)



1. Select a location and select Lite plan.
2.
curl -X GET "$vpc_api_endpoint/v1/volumes?version=2022-12-09&generation=2" -H "Authorization: $iam_token"

A successful response looks like the following example. This example shows three data volumes. The first in the list is attached to an instance.

{
"first": {
"href": "https://us-south.iaas.cloud.ibm.com/v1/volumes?limit=50"
},
"limit": 50,
"volumes": [
{
"active": true,
"bandwidth": 128,
"busy": false,
"capacity": 100,
"created_at": "2022-12-09T06:26:17Z",
"crn": "crn:...]",
"encryption": "provider_managed",
"health_reasons": ],
"health_state": "ok",
"href": "https://us-south.iaas.cloud.ibm.com/v1/volumes/ccbe6fe1-5680-4865-94d3-687076a38293",
"id": "ccbe6fe1-5680-4865-94d3-687076a38293",
"iops": 1000,
"name": "my-volume-1",
"profile": {
"href": "https://us-south.iaas.cloud.ibm.com/v1/volume/profiles/general-purpose",
"name": "general-purpose"
},
"resource_group": {
"href": "https://resource-controller.cloud.ibm.com/v2/resource_groups/4bbce614c13444cd8fc5e7e878ef8e21",
It is defined through the catalog. You can define several service instances based on the same or different plans, which offer access to different quantum computing backends. See [Qiskit Runtime plans](https://cloud.ibm.com/docs/quantum-computing?topic=quantum-computing-plans) for more details.
* Project: A grouping unit that enables users to work on the same resources. This tutorial uses two projects; ml and finance. See [Hierarchical project structures](https://cloud.ibm.com/docs/quantum-computing?topic=quantum-computing-considerations-orgnest-org) for more information.

This project is not related to the "project" concept in IBM Quantum Platform.









 Step 1: Plan your setup 

Before you set up Qiskit Runtime for your organization, you need to make these decisions:



* How are user identities defined? You can set up IBM Cloud users, users from another IDP, or both.



* If you are using a different IDP, does the Cloud administrator or the IDP administrator assign users to project resources?
* If the IDP administrator assigns users to projects, you need a string to be used as a key, such as project (which this tutorial uses) for project comparisons.



* What are the projects and which service instances will belong to each? You must plan your project names carefully.



* Do not make project names substrings of another. For example, if you use ml and chemlab for project names, then later you set up a project match for ml, it triggers on both values, accidentally granting more access than expected. Instead, use unique names such as ml and chem-lab. Alternatively, use prefix or suffix values to avoid such unintended substring matches.
* Using naming conventions, along with prefix or suffix values can help you easily allow access to several projects.
* Quantum experiments (jobs) belong to service instances, and users that have access to an instance can see its jobs.
* Service instances can be based on different plans, allowing access to different backends like real devices or simulators.
This data store, also known as the directory, contains information about AD objects. These objects typically include shared resources such as servers, volumes, printers, and the network user and computer accounts.
* Replication - A service that distributes directory data across a network. All domain controllers in a domain participate in replication and contain a complete copy of all directory information for their domain. Any change to directory data is replicated to all domain controllers in the domain.
* Organizational units - OUs can be used to form a hierarchy of containers within a domain. OUs are used to group objects for administrative purposes such as the application of Group Policy or delegation of authority. Control (over an OU and the objects within it) is determined by the access control lists (ACLs) on the OU and on the objects in the OU. To facilitate the management of large numbers of objects, Active Directory Domain Services support the concept of delegation of authority. By using delegation, owners can transfer full or limited administrative control over objects to other users or groups. Delegation is important because it helps to distribute the management of large numbers of objects across a number of people who are trusted to perform management tasks.
* Functional levels - Functional levels determine the available Active Directory Domain Services domain or forest capabilities and the Windows® Server operating systems you can run on domain controllers in the domain or forest. However, functional levels do not affect which operating systems you can run on workstations and member servers that are joined to the domain or forest.







 Domain models 

Review the typical domain models available to a customer when they designate their domains either on-premises or in the IBM Cloud®.



* Single forest, single domain model.
* Single forest, multiple domains model.
* Multiple forests model.



For more information about forest and domain models, see [Designing the logical structure](https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/designing-the-logical-structure).



 Single forest, single domain model 

The single forest, single domain model is the easiest to administer and maintain. The design consists of a forest that contains a single domain. As it is the only domain, it is the forest root domain, and it contains all of the user and group accounts in the forest and the resource objects.
Related resources 

These resources are created by IBM Cloud® and by some IBM partners and customers.



 Papers and case studies 



* [Hybrid Cloud white paper](https://public.dhe.ibm.com/cloud/bluemix/network/direct-link/ibm-hybrid-cloud-whitepaper.pdf)

Abstract: Krystallize Technologies set up three different “real world” configurations that use the architecture that is described in this paper to design a dedicated, low-latency, private connection between cloud services and customer-provided network equipment, storage, and compute infrastructure that is durable and affordable.

This paper presents test results that show how a well-designed, well-executed, hybrid cloud architecture performs versus one that uses standard facilities, using connections and an architecture that is not optimized for application performance.



* Direct Link Dedicated Hosting solution features and benefits
* Reference solution for implementing NetApp private storage for IBM Cloud, colocated in a Digital Reality data center
* Results of performance tests conducted by Krystallize Technologies



* [Case Study for Direct Link and IBM Cloud](https://www.digitalrealty.com/case-studies/ibm-cloud)

Abstract: As IBM Cloud needed to expand its global data center footprint to meet burgeoning demand for its Infrastructure-as-a-Service offering, it needed the right data center partner to achieve the best available combination of power, space, and price.

Delivery systems that put extra steps between server deployments, cloud, and customers were too slow and indirect to deliver the low latency and high throughput IBM Cloud promises its customers.

Choosing to colocate at Digital Realty’s hub locations in APAC and the US puts IBM Cloud data centers right next door to Digital Realty’s world-class data center deployments, with direct dark fiber connection between them.







 Featured announcements 



* [InterCloud Offers Dedicated Connectivity to IBM Cloud](https://info.intercloud.com/intercloud-offers-dedicated-connectivity-to-ibm-cloud)

Discover how IBM Cloud and InterCloud are helping to enable a national railway company migrate business-critical applications to the cloud over a secure, dedicated connection.
Each view is a relational data structure that contains multiple data records. Each record is composed of values in columns that are defined by the view’s schema. To facilitate representing these models, which are custom and therefore have various schemas, a uniform JSON output schema is used.



* Each JSON object represents an Annotation Query Language (AQL) view.
* The name-and-value pairs in the JSON objects represent the names and values of the attributes in the view.
* The tuples in an AQL view are represented as an array of JSON objects, with one object for each tuple in the view.



The following table describes how AQL data types are represented in JSON syntax.



Advanced rules model JSON output schema

 AQL data type JSON syntax JSON example 

 Integer number 5 
 Float number 4.13 
 Boolean boolean true 
 Text string "some string" 
 Span object with the form {"text": String, "location": {"begin": Integer, "end": Integer}} { "text": "Jane", location": {"begin": 5, "end": 9} } 
 Special case: null value null null 
 List of Integer array of number values [ 1, 2, 3, 4, 5] 
 List of Float array of number values [ 4.13, 4.5 ] 
 List of Boolean array of boolean values [ true, true, false] 
 List of Text array of string values [ "some string", "another string" ] 
 List of Span array of objects with the form {"text":String, "location": {"begin": Integer, "end": Integer}} [{ "text":"Jane", "location": {"begin": 5, "end": 9} }, { "text":"...", "location": {"begin": 15, "end": 40} }] 
 Special case: empty List array with 0 elements [ ] 







 Advanced rules model limits 

The number of advanced rules models that you can define per service instance depends on your Discovery plan type.



Advanced rules model plan limits

 Plan Advanced rules models per service instance
<-- </section "id="section-examples-logpull-update" "> --><-- </section "id="section-log-retention-update" "> --><-- </section "id="section-log-retention-cli-ref" "> --><-- <section "id="section-metrics" "> --> Metrics Manipulate metrics by using the following metrics commands.<-- <section "id="section-firewall-event-analytics" "> --> ibmcloud cis firewall-event-analytics Retrieve a full log of firewall events. ibmcloud cis firewall-event-analytics DNS_DOMAIN_ID --dataset DATA_SET] --filter FILTER] --order FILTER_ORDER] --limit LIMIT_NUMBER] -i, --instance INSTANCE] --output FORMAT] ! !! ! ! ! ! ! ! !
<-- <section "id="section-firewall-event-analytics-options" "> --> Command options DNS_DOMAIN_ID: The ID of DNS domain. Required.--dataset: Requested dataset. The default value is firewallEventsAdaptiveGroups.
Use the following table to identify which datasets are included in your plan and the range of historical data you can query.

<-- <table> -->

Datasets included in your plan

 Dataset Trial / Standard Enterprise / Security / GLB 

 firewallEventsAdaptiveGroups 30 days 30 days 
 firewallEventsAdaptive 30 days 30 days 

<-- </table ""> -->

--filter: Filter events.
Presto overview 



 What is Presto? 

Presto is a distributed SQL query engine, with the capability to query vast data sets located in different data sources, thus solving data problems at scale.

Presto provides the ANSI SQL interface, which can be used for all data analytics and IBM® watsonx.data use cases. With this feature, you do not need to manage multiple query languages and interfaces to different databases and storage. Presto is designed for storage abstraction, which allows connections to any data source through its connectors.

watsonx.data uses version 0.279 of Presto.





 Presto server types 

A Presto installation includes three server types - Coordinator, Worker, and Resource manager. Following is a brief explanation of the server types. For more information about the server types, see [Presto concepts](https://prestodb.io/docs/current/overview/concepts.html) in Presto documentation.



* Coordinator - A coordinator is a server type in a Presto installation, which is responsible for parsing statements, planning queries, and managing Presto worker nodes. It is the brain of a Presto installation and is also the node to which a client connects to submit statements for execution. It is also responsible for fetching results from the workers and returning the results to the client.
* Worker - A worker is a server type in a Presto installation, which is responsible for running tasks and processing data. Worker nodes fetch data from connectors and exchange intermediate data with each other.
* Resource manager - The resource manager is a server type in presto, which aggregates data from all coordinator and workers and creates a global view of the Presto cluster.



The following connectors are supported in watsonx.data:



* [Iceberg](https://prestodb.io/docs/current/connector/iceberg.html)
* [Db2](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-db2_connector)
* [Netezza](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-netezza_connector)
[] | .id')
ibmcloud sl vs create --hostname $hostname --domain $domain --cpu $vcpu --memory $mem --datacenter $dc --os WIN_LATEST_64 --disk 100 --san --network 1000 --vlan-public $pubvlanid --vlan-private $privlanid --public-security-group $sgid





 Related links 



* [Cyber recovery with Veeam architecture overview](https://cloud.ibm.com/docs/vmwaresolutions/?topic=vmwaresolutions-veeam-cr-sa-overview)
Each storage node has multiple paths to its own Solid-State Drives and its partner node's SSDs as well. This configuration protects against path failure, and also controller failure because the node can still access its partner's disks seamlessly. For more information, see [Availability and Durability of Block Storage for Classic](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-storageavailability).
* How can I identify a Block Storage for Classic volume from my OS?

Various reasons exist for why you would want to look up the LUN ID of the attached storage volumes on the Compute host. For example, you might have multiple storage devices that are mounted on the same host with the same volume sizes. You want to detach and decommission one of them. However, you are not sure how to correlate what you see on your Linux® host with what you see in the console. Another example might be that you have multiple Block Storage for Classic volumes that are attached to an esxi server. You want to expand the volume size of one of the LUNs, and you need to know the correct LUN ID of the storage to do that. For OS-specific instructions, click one of the following links.



* [Viewing LUN information in Linux®](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-identifyLUNidentifyLUNLin)
* [Viewing LUN information in Windows®](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-identifyLUNidentifyLUNWin)
* [Viewing LUN information in VMWare®](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-identifyLUNidentifyLUNVMware)



* Can I get storage performance metrics (IOPS or latency) from the Support teams?

IBM Cloud® does not provide storage performance IOPS and latency metrics. Customers are expected to monitor their own Block Storage for Classic devices by using their choice of third-party monitoring tools.

The following examples are utilities that you might consider to use to check performance statistics.
For more information, see [Route filtering rules](https://cloud.ibm.com/docs/dl?topic=dl-filter-routes&interface=uiroute-filtering-rules).

--help|-h
: Get help on this command. Optional.

--output value
: Specify whether you want the output displayed in JSON format. Currently, json is the only supported format. Optional.





 Examples 



* ibmcloud dl erfu beec5de0-a1c0-4730-8888-2ce4a3020ec8 c3df351b-ff64-4444-ba64-92cb7622dfd6 --action permit
* ibmcloud dl erfu beec5de0-a1c0-4730-8888-2ce4a3020ec8 c3df351b-ff64-4444-ba64-92cb7622dfd6 --le 28









 ibmcloud dl gateway 

View details of a specific gateway.

ibmcloud dl gateway|gw GATEWAY_ID [-–help|-h] [--output format]



 Command options 

GATEWAY_ID
: Specify the ID of the gateway.

--help|-h
: Get help on this command. Optional.

--output value
: Specify whether you want the output displayed in JSON format. Currently, json is the only supported format. Optional.





 Examples 



* ibmcloud dl gateway a771366f-2c8c-49f6-a23b-9d49fad035a3
* ibmcloud dl gw a771366f-2c8c-49f6-a23b-9d49fad035a3 --output json









 ibmcloud dl gateway-change-approve 

Approve gateway change request.
* [Raft](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-raft)
* [SDK](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-sdk)
* [Service credentials](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-service-credentials)
* [Service discovery](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-sd)
* [Shim](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-shim)
* [SignCert](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-sign-cert)
* [Smart contracts](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-smart-contracts)
* [Smart contract definition](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-smart-contract-def)
* [Smart contract endorsement policy](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-transaction-ep)
* [Smart contract package](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-smart-contract-pkg)
* [State database](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-state-database)
* [Transaction](https://cloud.ibm.com/docs/blockchain?topic=blockchain-glossaryglossary-transaction)
--extras '{"hardware":[{"hostname":"lhbr01","domain":"partner.ibmcloud.local","primaryBackendNetworkComponent": {"networkVlan": {"id": 3127648}}}],"storageGroups":[{"arrayTypeId": 2,"arraySize": 1000,"hardDrives": 0,1],"partitionTemplateId": 1},{"arrayTypeId": 4,"arraySize": 12000,"hardDrives": 2,3,4,5,6,7,8,9]}],"sshKeys": [{"sshKeyIds": 2097510]}]}' 
--billing monthly 
--verify
Show more

The following command is the example of a sample order:

ibmcloud sl order place 
2U_DUAL_INTEL_XEON_PROCESSOR_SCALABLE_FAMILY_12_DRIVES 
DALLAS10 
INTEL_INTEL_XEON_5120_2_20,
OS_RHEL_8_X_64_BIT_PER_PROCESSOR_LICENSING,
RAM_64_GB_DDR4_2133_ECC_NON_REG,
DISK_CONTROLLER_RAID,
HARD_DRIVE_1_00_TB_SATA_2,HARD_DRIVE_1_00_TB_SATA_2,
HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,HARD_DRIVE_2_00_TB_SATA_2,
BANDWIDTH_0_GB,
10_GBPS_REDUNDANT_PRIVATE_NETWORK_UPLINKS,
REBOOT_KVM_OVER_IP,
1_IP_ADDRESS,
> * [ibm_is_ike_policies](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_ike_policies)<br> * [ibm_is_ike_policy](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_ike_policy)<br> * [ibm_is_image](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_image)<br> * [ibm_is_images](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_images)<br> * [ibm_is_instance](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_instance)<br> * [ibm_is_instance_disk](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_instance_disk)<br> * [ibm_is_instance_disks](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/is_instance_disks)<br