vania-janet/multiturn-rag-retrieval-data huggingface dataset:
BI.S4.H8.6TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2433)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2434)<br><br><br> 
 BI.S4.H8.6TB RAM + 12TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2512)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2507)<br><br><br> 
 BI.S3.H2.192 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2019)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2256)<br><br><br>
* ALIAS NAME: specify the data engine endpoint for your region. For more information on the currently supported data engine endpoints, see [Data engine endpoints](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewendpoints).
* THRIFT URL: specify the region-specific thrift URL. For example, thrift://catalog.us.dataengine.cloud.ibm.com:9083. For more information about the applicable endpoints(thrift), see [Thrift endpoint](https://cloud.ibm.com/docs/sql-query?topic=sql-query-hive_metastorehive_compatible_client).
* CRN-DATA-ENGINE-INSTANCE: specify the crn of the data engine instance.



Make sure that you select the standard aliases.





 Read Data from table using simpler Convenience API 

If you want to do a quick test of the Hive metastore by specifying IBM Cloud Data Engine connection API in your application, you can use the convenience API shown in the following PySpark example.

In this example, there is no need to pass any IBM Cloud Data Engine Hive metastore parameters to your application. The call to SparkSessionWithDataengine.enableDataengine initializes the connections to IBM Cloud Data Engine without the additional IBM Cloud Data Engine Hive metastore parameters.

dataengine-job-convenience_api.py:

from dataengine import SparkSessionWithDataengine
from pyspark.sql import SQLContext
import sys
 from pyspark.sql import SparkSession
import time

def dataengine_table_test(spark,sc):
  tablesDF=spark.sql("SHOW TABLES")
  tablesDF.show()
  statesDF=spark.sql("SELECT * from COUNTRIESCAPITALS");
  statesDF.show()

def main():
  if __name__ == '__main__':
    if len (sys.argv) < 2:
        exit(1)
    else:
BI.S4.H2.768 IBM Cloud Bare Metal (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2188)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2249)<br> * [Business One Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2204)<br><br><br> 
 BI.S4.H2.1500 IBM Cloud Bare Metal (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2253)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2191)<br><br><br>
BI.S4.H8.6000 IBM Cloud Bare Metal (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2303)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2305)<br><br><br> 
 BI.S4.H8.12000 IBM Cloud Bare Metal (Local SSD) <br><br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2302)<br> * [OLTP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2304)<br><br><br> 
 BI.S4.H2.384GB RAM + 1.5TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2513)<br><br><br> 
 BI.S4.H2.768GB RAM + 1.5TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2502)<br><br><br>
For more information, see [Data source requirements](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesrequirements).

Before you implement a custom connector, you need to know the following information about the data source:



* The data source's network location (server name or address, including port, or URL, including port)
* The data source's authentication method and security credentials
* The path or paths on the data source that the connector needs to crawl
* The connection method or protocol that the data source supports







 Designing a custom connector 

A custom connector needs the following capabilities:



* Configuring a crawler.



* Configuring all settings that are required to connect to the data source.
* Discovering a crawl space on the data source. At least one crawl space is required.



* Crawling documents.



* Crawling the documents on each data set.
* Adding Access Control List (ACL) information to each document.



* Retrieving ACL information for the username that authenticates to the data source.



These capabilities can be implemented by using the interfaces and methods that are described in [Developing custom connector code](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-dev).





 Custom connector limitations 

Observe the following notes and warnings when you implement a custom connector.



* Custom connectors do not support the following features:



* Synchronization settings
* Filtering documents based on user access at query time. (At crawl and index time, only documents that the current user has the right to access are returned.)
* The required and hidden validation settings. They are ignored when the connector is displayed in Discovery
* The use of <condition /> tags in the definition file. These tags are currently ignored.



* When you use the example connector code in the current release, Discovery does not collapse and group authentication settings for the custom connector's properties.
Notebooks 

Jupyter Notebooks are web-based environments for interactive computing. You can run small pieces of code that process your data, and you can immediately view the results of your computation. Notebooks include all of the building blocks that you need to work with data.

Check out the following Notebooks to get familiar with IBM Cloud® Data Engine:



*  Learn how to use the Data Engine API to run SQL statements in a programmatic way for data analytics and ETL, by using the ibmcloudsql Python library in [Using IBM Cloud Data Engine](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/e82c765fd1165439caccfc4ce8579a25?context=cpdaas). The Notebook also demonstrates the library features and how to combine Data Engine with visualization libraries, such as PixieDust.
*  Learn how to combine Data Engine with visualization libraries, such as PixieDust, folium, or matplotlib to explore particulate matter data in [Explore particulate matter data by using Data Engine](https://eu-gb.dataplatform.cloud.ibm.com/exchange/public/entry/view/5d686c16d14491f4c3997b67fe11506d).
* With monthly billing, the calculation for the price is pro-rated from the date of creation to the end of the billing cycle and billed immediately. If a block LUN is deleted before the end of the billing cycle, the difference is not refunded. Monthly billing is a good choice for storage that is used in production workloads that use data that needs to be stored and accessed for long periods of time (month or longer).



4. Enter your storage size in the Size field.
5. Select the size of the Snapshot space from the list.

This space is in addition to your usable space. For snapshot space considerations and recommendation, read [Ordering Snapshots](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-orderingsnapshots).
6. Choose your OS Type from the list.

This selection is based on the operating system that your host is running on and it cannot be modified later. For example, your server is Ubuntu or RHEL, select Linux®. If your host is a Windows® 2012 or Windows® 2016 server, select the Windows® 2008+ option from the list. For more information about various Windows® options, see the [FAQ](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-block-storage-faqswindowsOStypes).
7. Select your IOPS profile. You can choose between the predefined values of Endurance (Tiers) or enter your custom IOPS value for Performance.



* 0.25 IOPS per GB is designed for workloads with low I/O intensity. These workloads are typically characterized by having a large percentage of data inactive at a time. Example applications include storing mailboxes or departmental level file shares.
* 2 IOPS per GB is designed for most general-purpose usage. Example applications include hosting small databases that are backing web applications or virtual machine disk images for a hypervisor.
* 4 IOPS per GB is designed for higher-intensity workloads. These workloads are typically characterized by having a high percentage of data active at a time.
For data center location, click the Edit icon ![Edit icon](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/icons/edit-tagging.svg) and select the geography, data center, and pod where the instance is to be hosted.
2. Select the CPU model:



* For Cascade Lake servers, select the CPU model and the RAM size.
* For SAP-certified Cascade Lake servers, choose one of the preset configurations.



3. Specify the number of bare metal servers.



6. If you selected the VMware vSAN component, complete the vSAN™ storage configuration.



* If you want more storage, select the High performance with Intel Optane checkbox.
* Specify the disk types for the capacity and cache disks, and the number of disks.
* By default, the Enable vSAN deduplication and compression checkbox is selected. If you do not want to enable vSAN deduplication and compression, clear the checkbox.



7. Complete the network interface settings:



1. Enter the hostname prefix and domain name.
2. Select the network setting of either Public and private network or Private network only.
3. Select the uplink speed. The 25 Gb option is available only for specific pods and data center locations.
4. Select the network interface that you want to use.



* If you want to order new public and private VLANs, click Order new VLANs.
* If you want to reuse the existing public and private VLANs when they are available, click Select existing VLANs and specify the VLANs and optionally the subnets.



5. If you are ordering public VLANs, specify whether to apply the FortiGate Physical Appliance 300 Series HA Pair to secure your cloud environment.



8. In the Summary pane, verify the cluster configuration and the estimated price.



* To save the configuration as a template without placing an order, click Save configuration.
View the complete list of entity types and subtypes [here](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-entity-type-systems). For example:

Input

> text: "IBM is an American multinational technology company headquartered in Armonk, New York, United States, with operations in over 170 countries."

Response

> IBM: Company
> Armonk: Location
> New York: Location
> United States: Location





 Keywords 

Search your content for relevant keywords. For example:

Input

> url: "[http://www-03.ibm.com/press/us/en/pressrelease/51493.wss](http://www-03.ibm.com/press/us/en/pressrelease/51493.wss)

Response

> Australian Open
> Tennis Australia
> IBM SlamTracker analytics





 Metadata 

For HTML and URL input, get the author of the webpage, the page title, and the publication date. For example:

Input

> url: "[https://www.ibm.com/blogs/think/2017/01/cognitive-grid/](https://www.ibm.com/blogs/think/2017/01/cognitive-grid/)

Response

> Author: Stephen Callahan
> Title: Girding the Grid with Cognitive Computing - THINK Blog
> Publication date: January 31, 2017





 Relations 

Recognize when two entities are related, and identify the type of relation. For example:

Input

> text: "The Nobel Prize in Physics 1921 was awarded to Albert Einstein."

Response

> awardedTo" relation between "Noble Prize in Physics" and "Albert Einstein
> timeOf" relation between "1921" and "awarded





 Semantic Roles 

Parse sentences into subject-action-object form, and identify entities and keywords that are subjects or objects of an action. For example:

Input

> text: "In 2011, Watson competed on Jeopardy!"

Response

> Subject: Watson
> Action: competed
It provides data orchestration and metadata management across heterogeneous data sources.

The SAP Data Hub is a fundamental technology in the SAP strategy. Part of the digital platform, SAP Data Hub is a connector and processing layer on which to build new applications. By enabling connections across the whole SAP product portfolio, SAP Data Hub helps the business build their Intelligent Enterprise.



 Common Scenarios 



Table 1. Common scenarios for SAP Data Hub

 Use Case Details 

 IOT Sensor Data Ingestion Integrate and transform data streams from any sensor or device into data that can be used by ERP, AI/ML, and other Big Data services 
 Big Data, Machine Learning, Artificial Intelligence Move your data science projects from PowerPoint to operational applications, gleaning real insight from what you’re already collecting 
 Data Warehousing Make it possible to add disparate data streams into your data warehouse, and transform it from an archival reporting system to a core business value 
 Master Data and Governance Automate your metadata tagging and discovery, understand what data is coming from where, and control who can see the data based on security and privacy standards
](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_storage_pool_capacity)<br> * [ibm_pi_storage_pools_capacity](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_storage_pools_capacity)<br> * [ibm_pi_storage_type_capacity](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_storage_type_capacity)<br> * [ibm_pi_storage_types_capacity](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_storage_types_capacity)<br> * [ibm_pi_system_pools](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_system_pools)<br> * [ibm_pi_tenant](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_tenant)<br> * [ibm_pi_volume](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_volume)<br> * [ibm_pi_volume_flash_copy_mappings_html](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_volume_flash_copy_mappings_html)<br> * [ibm_pi_volume_group](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/data-sources/pi_volume_group)<br> * [ibm_pi
* [Dynamic Content Acceleration container](https://cloud.ibm.com/docs/CDN?topic=CDN-dynamic-content-acceleration-container)
* [Geoblocking class](https://cloud.ibm.com/docs/CDN?topic=CDN-geoblocking-class)
* [Hotlink Protection class](https://cloud.ibm.com/docs/CDN?topic=CDN-hotlink-protection-class)
* [Input container](https://cloud.ibm.com/docs/CDN?topic=CDN-input-container)
* [Mapping container](https://cloud.ibm.com/docs/CDN?topic=CDN-mapping-container)
* [Metrics container](https://cloud.ibm.com/docs/CDN?topic=CDN-metrics-container)
* [Modify Response Header container](https://cloud.ibm.com/docs/CDN?topic=CDN-modify-response-header-container)
* [Path (Origin) container](https://cloud.ibm.com/docs/CDN?topic=CDN-path-origin-container)
* [Token authentication container](https://cloud.ibm.com/docs/CDN?topic=CDN-token-auth-container)
* [Vendor container](https://cloud.ibm.com/docs/CDN?topic=CDN-vendor-container)
* [Purge Group container](https://cloud.ibm.com/docs/CDN?topic=CDN-purge-group-container)
* [Purge History container](https://cloud.ibm.com/docs/CDN?topic=CDN-purge-history-container)



* [Activity Tracker events for CDN](https://cloud.ibm.com/docs/CDN?topic=CDN-at_events)
Enter mycloud.com in the Domain field. Domain is the identification string that defines administrative control within the internet. Click Information for formatting specifics.
7. Billing defaults to Monthly. Currently, 1-year contract and 3-year contract are not available for SAP-certified servers.
8. The data centers displayed under Location depend on product availability within a particular data center. Leave the default Location of NA South DAL10-Dallas.
9. Click All servers > SAP certified.





 Configuring your server 

Select your SAP-certified server and OS.



1. Select CPU Model BI.S3.NW32 (OS Options). For more information about deciphering the server names, see [Provisioning your Bare Metal Servers using the IBM Cloud console](https://cloud.ibm.com/docs/sap?topic=sap-nw-iaas-offerings-profiles-intel-bm).
2. RAM defaults to a predefined value based on your server selection and cannot be changed.
3. Enter an optional public key for your SSH key, which you can use to log in to your server after provisioning is done. The default is None.
4. Choose Microsoft as your Image (OS) and select 2016 Standard (64 bit)-HVM.

If you're bringing your own license (BYOL) for your OS, select No OS. For more information, see [Bring your own license](https://cloud.ibm.com/docs/sap?topic=sap-quickstudy-bm-netweaver-winsbyol-32GB).







 Adding storage disks 



1. Under Type, select RAID 10.
2. Disks, Hot Spare, and Disk Media have default values that are based on your selection. Select a Disk Size that covers the total amount of storage you need.
3. Click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/icons/action-menu-icon.svg) > Advanced configuration and leave Controller cleared. Click OK.
* [VMware vSphere overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_vsphereclusteroverview)
* [Getting started with IBM Cloud Direct Link (2.0)](https://cloud.ibm.com/docs/dl?topic=dl-get-started-with-ibm-cloud-dl)
* [Configuring EVPN](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-D8186088-6C8F-4553-B859-B9499D9FB559.html)
BI.S4.H8.3TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2511)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2520)<br><br><br> 
 BI.S4.H8.3TB RAM + 12TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2518)<br><br><br> 
 BI.S4.H8.6TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2433)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2434)<br><br><br>
BI.S4.H2.192 IBM Cloud Bare Metal (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2190)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2252)<br> * [Business One Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2200)<br><br><br> 
 BI.S4.H2.384 IBM Cloud Bare Metal (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2189)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2250)<br> * [Business One Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2203)<br><br><br>
* [OpenShift Data Foundation using local disks](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-odf-local)
* [OpenShift Data Foundation for remote devices](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-odf-remote)



* VMware storage templates



* [VMware Block Container Storage Interface (CSI) Driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-vsphere-csi-driver)



* [Storage class reference ](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-ref)



* [Setting up Satellite as a Secure Gateway for on-prem solutions](https://cloud.ibm.com/docs/satellite?topic=satellite-sg-usecase)
* Enhancing security



* Managing access



* [Managing access overview](https://cloud.ibm.com/docs/satellite?topic=satellite-iam)
* [Assigning access with IBM Cloud IAM](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-assign-access)
* [Platform access roles](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-platform-access)
* [Service access roles](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-service-access)
* [API keys in IBM Cloud ](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-api-key)
* [Common permissions in other cloud providers](https://cloud.ibm.com/docs/satellite?topic=satellite-iam-common)



* [Learning about Satellite architecture, workload isolation, and dependencies](https://cloud.ibm.com/docs/satellite?topic=satellite-service-architecture)
* [Security and compliance](https://cloud.ibm.com/docs/satellite?topic=satellite-compliance)
Some storage solutions, such as file and block storage, are data center-specific and can't be shared across zones in a multizone cluster setup.

If you want to make your data accessible across zones or regions, make sure to consult your legal department to verify that your data can be stored in multiple zones or a different country.

Understand other storage characteristics that impact your choice.
: Consistency: The guarantee that a read operation returns the latest version of a file. Storage solutions can provide strong consistency when you are guaranteed to always receive the latest version of a file, or eventual consistency when the read operation might not return the latest version. You often find eventual consistency in geographically distributed systems where a write operation first must be replicated across all instances.
: Performance: The time that it takes to complete a read or write operation.
: Durability: The guarantee that a write operation that is committed to your storage survives permanently and does not get corrupted or lost, even if gigabytes or terabytes of data are written to your storage at the same time.
: Resiliency: The ability to recover from an outage and continue operations, even if a hardware or software component failed. For example, your physical storage experiences a power outage, a network outage, or is destroyed during a natural disaster.
: Availability: The ability to provide access to your data, even if a data center or a region is unavailable. Availability for your data is usually achieved by adding redundancy and setting up failover mechanisms.
: Scalability: The ability to extend capacity and customize performance based on your needs.
: Encryption: The masking of data to prevent visibility when data is accessed by an unauthorized user.



 Non-persistent storage 

Virtual Private Cloud

Classic infrastructure

Satellite

You can use non-persistent storage options if your data is not required to be persistently stored or if you want to unit-test your app components. The following image shows available non-persistent data storage options in IBM Cloud Kubernetes Service.

Zoom

!