vania-janet/multiturn-rag-retrieval-data huggingface dataset:
IBM Cloud Data Engine is the key component in the Stream Landing approach. It is the service that connects to Event Streams and copies the data to IBM Cloud Object Storage.

[Parquet](https://parquet.apache.org/docs/) is an open source file format for nested data structures in a flat columnar format. Compared to the traditional approach where data is stored in rows, Parquet is more efficient in terms of storage and performance.



1. In your browser, navigate to the [resource list](https://cloud.ibm.com/resources) and under Integration, click on Event Streams log-analysis-es service.
2. Select Topics from the navigation pane on the left.
3. Select the context menu (three vertical dots) for your topic webserver and click Create stream landing configuration.

Zoom

![Event Streams topics](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution31/event_streams_topics.png)

Event Streams topics
4. Click Start and select the log-analysis-cos service. Click Next.
5. Select the ABC-log-analysis Object Storage bucket and click Next.
6. Select the Data Engine log-analysis-sql service and click Next.
7. Configure how you want your topic data to be streamed to Object Storage:



1. Prefix for objects added to Object Storage bucket: logs-stream-landing
2. Create a new Service ID : logs-stream-landing-service-id
3. Select the log-analysis-kp Key Protect service.
4. Click Start streaming data.





You now see the status Queued for your topic. It may take up to 5 minutes until the streaming job is fully dispatched and up and running. You will see the status switch to Running at that point. In the context menu, you find a new option called View stream landing configuration.





 Using kcat with Event Streams
Existing instances still work but will be fully deprecated on 31 October.





 May 2022 

Rebranding
: IBM Cloud SQL Query was rebranded to IBM Cloud Data Engine.

Hive
: Data Engine provides an external [Hive metastore (HMS) service](https://cloud.ibm.com/docs/sql-query?topic=sql-query-hive_metastore).





 November 2021 

Add columns to Catalog tables
: You can add columns to existing Catalog tables with the newly supported ALTER TABLE ... ADD COLUMNS statement.





 July 2021 

Stream landing tutorial
: A detailed [getting started tutorial](https://www.ibm.com/cloud/blog/stream-landing-from-event-streams-kafka-service-to-ibm-cloud-data-lake-on-object-storage) for stream landing with Data Engine is now available.

New region for stream landing
: The stream landing capability is now also available in Frankfurt, in addition to Dallas.





 June 2021 

Stream landing support
: Data Engine now supports stream landing that enables you to stream your data in real time from a topic to a bucket of your choice. This capability enables efficient analytics on the new objects created.

Connect to data lakes with Cloud Pak for Data
: IBM Cloud Pak® for Data now comes with an integrated connector to Data Engine that allows to connect to cloud data lakes and import data assets into projects and catalogs in Cloud Pak for Data. For more information, see [Connecting to a Cloud Data Lake with IBM Cloud Pak for Data](https://www.ibm.com/cloud/blog/connecting-to-a-cloud-data-lake-with-ibm-cloud-pak-for-data).





 December 2020 

Supported regions
: Data Engine is available in Chennai, India. When you provision new instances, you can select whether it is being provisioned in Dallas, Frankfurt, or Chennai.

IBM Cloud Object Storage
: IBM Cloud Object Storage web console discovers SQL-queryable objects and folders and directly starts the Data Engine web console with a prefilled SQL statement for seamless interactive data exploration.





 November 2020 

Modify location of Hive partitions
Ingesting data from data lakes 

If you plan on regularly querying your data, load it into Netezza Performance Server first to get the best performance benefits.



 Before you begin 

In the examples, the publicly available [New York taxi trip record data](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) for yellow taxis in January 2022 is used. To follow this example, make sure that the data is in an accessible S3 bucket.





 1. Create an external data source. 

External datasources allow an administrator to grant access to S3 without providing the keys directly to a user.

a) Set ENABLE_EXTERNAL_DATASOURCE.

set ENABLE_EXTERNAL_DATASOURCE = 1;

b) Create an external data source.

create EXTERNAL DATASOURCE 'DATA SOURCE' on 'REMOTE SOURCE'
using (
ACCESSKEYID 'ACCESS KEY ID' SECRETACCESSKEY 'SECRET ACCESS KEY' BUCKET 'BUCKET' REGION 'REGION'
);

Example:

create EXTERNAL DATASOURCE EXAMPLEDATALAKE on AWSS3 
using (
ACCESSKEYID 'XXXX' SECRETACCESSKEY 'XXXX' BUCKET 'exampledatalakebucket' REGION 'US-EAST-1'
);

For more information, see [CREATE EXTERNAL DATASOURCE command](https://www.ibm.com/docs/en/netezza?topic=tables-create-external-datasource-command).





 2. Create an external table for the data from a data lake. 

After you created an external data source, you can create an external table that accesses the yellow taxi data from January 2022.

Ensure that you have the necessary privileges as described in [Privileges for creating external tables](https://www.ibm.com/docs/en/netezza?topic=et-create-external-table-command-2).

create EXTERNAL table 'TABLE NAME' on 'DATA SOURCE'
using ( 
DATAOBJECT ('DATA OBJECT') FORMAT 'PARQUET' 
);
API 

To get you started with the IBM Cloud® Data Engine API, the [Using IBM Cloud Data Engine Jupyter Notebook](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/e82c765fd1165439caccfc4ce8579a25?context=cpdaas) demonstrates an end-to-end cycle. It starts with setting up libraries and configuring Data Engine, all the way to visualizing results by using PixieDust.
* [Classifier limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-classifierclassifier-limits)
* [Collection limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-limits)
* [Dictionary limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionarydictionary-limits)
* [Document limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-doc-limits)
* [Entity extractor limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-limits)
* [Machine Learning model limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-mlmachinelearning-limits)
* [Pattern limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-patternpatterns-limits)
* [Project limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsprojects-limits)
* [Query limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-conceptsquery-limits)
* [Regular expression limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-regexregex-limits)
* [SDU limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldssdu-limits)



The following limits apply only to Content Mining project types:



* [Document classifier limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-doc-classifierdoc-classifier-limits)
ux2d-4x112 IBM Cloud Virtual Server -- 4 112 4,312 Intel Cascade Lake SP 28.00 -- -- VPC 
 BI.S3.NW192 (VMware) IBM Cloud for VMware 36 72 192 70,965 Intel Skylake SP 2.67 -- -- Classic 
 BI.S3.NW384 (VMware) IBM Cloud for VMware 36 72 384 71,487 Intel Skylake SP 5.33 -- -- Classic 
 BI.S3.NW768 (VMware) IBM Cloud for VMware 36 72 768 71,667 Intel Skylake SP 10.67 -- -- Classic 
 BI.S4.NW192 (VMware) IBM Cloud for VMware 32 64 192 74,223 Intel Cascade Lake SP 3.00 -- -- Classic 
 BI.S4.NW384 (VMware) IBM Cloud for VMware 32 64 384 76,617 Intel Cascade Lake SP 6.00 -- -- Classic 
 BI.S4.NW768 (VMware) IBM Cloud for VMware 40 80 68 101,547 Intel Cascade Lake SP 9.60 -- -- Classic 
 BI.S4.NW1500 (VMware) IBM Cloud for VMware 56 112 1,536 132,498 Intel Cascade Lake SP 13.71 -- -- Classic 
 BI.S4.NW3000 (VMware) IBM Cloud for VMware 56 112 3,072 121,614 Intel Cascade Lake SP 27.43 -- -- Classic 



The profiles marked with * asterisk, are not listed on the SAP HANA Hardware Directory by SAP but are certified for running SAP HANA production systems. The directory lists the smallest, median, and largest within each profile family. This action has been taken by SAP to avoid too many records, as the scalability of IBM POWER hardware enables significantly more granular sizing. See [SAP Note 2947579 - SAP HANA on IBM Power Virtual Servers](https://launchpad.support.sap.com//notes/2947579) for more detail.





 Do you have a full list of Profile specifications for SAP HANA scale-out?
Enter mycloud.com in the Domain field. Domain is the identification string that defines administrative control within the internet. Click Information for formatting specifics.
7. Billing defaults to Monthly. Currently, 1-year contract and 3-year contract are not available for SAP-certified servers.
8. The data centers displayed under Location depend on product availability within a particular data center. Select NA East TOR01-Toronto.
9. Click All servers > SAP certified.





 Configuring your server 



1. Select CPU Model BI.S3.NW32 (OS Options). For more information about how to decipher the server names, see [Provisioning your Bare Metal Servers using the IBM Cloud console](https://cloud.ibm.com/docs/sap?topic=sap-nw-iaas-offerings-profiles-intel-bm).
2. RAM defaults to a predefined value based on your server selection and cannot be changed.
3. Enter an optional public key for your SSH key, which you can use to log in to your server after the server is provisioned. The default is None.
4. Select Red Hat as your Image (OS). The default is 7.x (64 bit).

If you're bringing your own license (BYOL) for your OS, select No OS. For more information, see [Bring your own license](https://cloud.ibm.com/docs/sap?topic=sap-quickstudy-bm-netweaver-rhelbyol-32GB-rhel).







 Adding storage disks 



1. Under Type, select RAID 10.
2. Disks, Hot Spare, and Disk Media have default values. Select a Disk size that covers the total amount of storage you need.
3. Click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/icons/action-menu-icon.svg) > Advanced configuration and leave Controller cleared. Click OK.







 Network interface 



1.
* [Digital Realty and IBM Cloud Direct Link - Built for Enterprise Cloud](https://www.ibm.com/cloud/blog/announcements/digital-realty-ibm-cloud-dl-expand-network), by Paul Hertzfeldt
* [PacketFabric and IBM Cloud Enable Secure Enterprise Connections](https://www.ibm.com/cloud/blog/announcements/packetfabric-ibm-enable-secure-enterprise-connections), by Paul Hertzfeldt
* [AT TOKYO Connects to IBM Cloud to Support Mission-Critical Applications](https://www.ibm.com/cloud/blog/announcements/tokyo-connects-ibm-cloud-support-mission-critical-applications), by Paul Hertzfeldt
* [Improve Digital Experiences with SES Networks and IBM](https://www.ibm.com/cloud/blog/improve-digital-experiences-with-ses-networks-and-ibm), by Paul Hertzfeldt







 Papers and case studies 



* [Hybrid Cloud white paper](https://public.dhe.ibm.com/cloud/bluemix/network/direct-link/ibm-hybrid-cloud-whitepaper.pdf)

Abstract - Krystallize Technologies set up three different "real world" configurations that use the architecture that is described in this paper to design a dedicated, low-latency, private connection between cloud services and customer-provided network equipment, storage, and compute infrastructure that is durable and affordable.

This paper presents test results that show how a well-designed, well-executed, hybrid cloud architecture performs versus one that uses standard facilities, using connections and an architecture that is not optimized for application performance.



* Reference solution for implementing NetApp private storage for IBM Cloud, colocated in a Digital Realty data center
* Results of performance tests conducted by Krystallize Technologies



* [Case Study for Direct Link and IBM Cloud](https://www.digitalrealty.com/case-studies/ibm-cloud)

Abstract - As IBM Cloud needed to expand its global data center footprint to meet burgeoning demand for its Infrastructure-as-a-Service offering, it needed the right data center partner to achieve the best available combination of power, space, and price.
: The JSON file or JSON string used to describe a GLB pool. Required.



* The optional fields are name, origins, description, healthy_origins_threshold, enabled, monitor, notification_channel, healthcheck_region, healthcheck_subnets:



* name: The name of the load balancer pool.
* origins: The list of origins within this pool.
* description: The descriptive text of the load balancer pool.
* healthy_origins_threshold: The minimum number of origins that must be healthy for this pool to serve traffic.
* enabled: Whether the load balancer pool is enabled. Valid values: true, false.
* monitor: The ID of the load balancer monitor to be associated to this pool.
* notification_channel: The notification channel.
* healthcheck_region: Health check region of VSIs. Valid values: us-south, us-east, eu-gb, eu-du, au-syd, jp-tok, br-sao, ca-tor.
* healthcheck_subnets: A list of health check subnet IDs of VSIs.





Sample JSON data:

{
"name": "us-pool",
"description": "application server pool in US",
"origins": [
{
"name": "us-app-dal10",
"description": "dallas origin 10",
"address": "1.1.1.1",
"enabled": true
},
{
"name": "us-app-dal12",
"description": "dallas origin 12",
"address": "2.2.2.2",
"enabled": true
}
],
"healthy_origins_threshold": 1,
"monitor": "f1aba936b94213e5b8dca0c0dbf1f9cc",
"enabled": true,
"notification_channel": "https://mywebsite.com/dns/webhook",
BI.S4.H8.3TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) 224 448 9216 Intel Cascade Lake SP 20.57 TDI OLAP/OLTP Classic 
 BI.S4.H8.3TB RAM + 12TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) 224 448 12288 Intel Cascade Lake SP 27.43 TDI OLTP Classic 
 BI.S4.H8.6TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) 224 448 15360 Intel Cascade Lake SP 34.29 TDI OLAP/OLTP Classic 
 BI.S4.H8.6TB RAM + 12TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) 224 448 18432 Intel Cascade Lake SP 41.14 TDI OLAP/OLTP Classic 
 -- -- -- -- -- -- -- -- -- -- -- 
 The following profiles are certified for SAP NetWeaver only - not SAP HANA The following profiles are certified for SAP NetWeaver only - not SAP HANA The following profiles are certified for SAP NetWeaver only - not SAP HANA The following profiles are certified for SAP NetWeaver only - not SAP HANA -- -- -- -- -- -- 
 BI.S3.NW32 IBM Cloud Bare Metal (Local SSD) 4 8 32 11,970 Intel Kaby Lake 4.00 -- -- Classic 
 BI.S3.NW64 IBM Cloud Bare Metal (Local SSD) 4 8 64 12,750 Intel Kaby Lake 8.00 -- -- Classic 
 BI.S3.NW192 IBM Cloud Bare Metal (Local SSD) 36 72 192 78,850 Intel Skylake SP 2.67 -- -- Classic 
 BI.S3.NW384 IBM Cloud Bare Metal (Local SSD) 36 72 384 79,430 Intel Skylake SP 5.33 -- -- Classic 
 BI.S3.NW768 IBM Cloud Bare Metal (Local SSD) 36 72 768 79,630 Intel Skylake SP 10.67 -- -- Classic
About DNS Services architecture and workload isolation 

Review the following sample architecture for IBM Cloud® DNS Services to learn about different isolation levels so that you can choose the solution that best meets your needs.



 Data isolation 

DNS Services is a multitenant service, sharing the infrastructure among all customers. DNS Services uses a Cloudant database to store customer-provided Domain Name System (DNS) configuration information. The data is encrypted at rest in Cloudant. Data is also encrypted in transit through SSL from the API endpoints within the control plane to the database.

DNS Services handles user-initiated DNS configuration requests in the control plane. In the control plane, DNS Services relies on IAM-based authorization and authentication so that only explicitly permitted users within the account can access the information. The IAM policies are enforced at the API level in a way that other user interfaces, such as UI and CLI, all inherit consistent access control policies. For information on which platform and service roles are allowed to take what action, see [Managing IAM and IBM Cloud DNS Services](https://cloud.ibm.com/docs/dns-svcs?topic=dns-svcs-iam).

Name resolution requests from the Virtual Private Cloud (VPC) are handled in the data plane. The user controls which VPC can access this DNS configuration data in the data plane. This can be specified using the control plane mechanism of adding a permitted network for the DNS zone. A user who has at least Operator level access to a VPC can add or remove that VPC as a permitted network for the DNS zone.

When a DNS query for a particular DNS zone is received by the resolver, it identifies the VPC where the DNS query originated and validates that it can access the DNS zone of the DNS query. If the VPC doesn't have access to the DNS zone, the DNS query is forwarded to a public resolver.

The public resolver that is used for forwarding is also in the IBM Cloud private underlay. A public resolver means that it resolves public DNS zones, although it is instantiated in the IBM Cloud private network.
For example, this store contains information about each VPC Image resource, such as its name, CRN, and creation date. However, it does not contain the image data itself, which is hosted on the storage devices. All communication between control plane services and the control plane data store is secure and encrypted. 
 Block Storage Each zone of each region contains a set of replicated storage devices, which host the data for your VPC volumes. Your data is always encrypted at rest, and is isolated to your account. In addition, you can use [Customer-managed encryption](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-about) and your own root keys for end-to-end encryption of your boot and data volumes. Regional buckets use Key Protect for data encrytion at rest. Storage data flows between storage devices and hypervisor nodes only, and does not flow to control nodes. 
 Snapshots Snapshots that you create of your volumes are stored in IBM Cloud® Object Storage and available for regional restoration. Your snapshots are protected using Key Protect for data encrytion at rest. Snapshot data flows between Object Storage and hypervisor nodes only, and does not flow to control nodes. 







 VPC dependencies 

The IBM Cloud® Virtual Private Cloud (VPC) depends upon various IBM Cloud integrated services for purposes such as:



* Hosting the internal microservices of the VPC service
* Integrating with IBM Cloud and its user interface
* Storing and backing up service data (including VPC resource metadata)
* Logging and auditing service events.



The following table lists the main dependencies of the VPC service and the purpose of each one:



Table 2. VPC dependencies

 Service Purpose 

 IBM Cloud® Identity and Access Management Provides authentication and authorization for all VPC requests. For more information about required IAM permissions, see [Required permissions for VPC resources](https://cloud.ibm.com/docs/vpc?topic=vpc-resource-authorizations-required-for-api-and-cli-calls).
To get a general idea of the types of entities that are recognized by the service, see [Entities](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-entity-types-version-2).

The following screen capture shows that the Entities enrichment recognizes the terms Systems of Government and King of Great Britain (among others) and tags them as entity mentions.

Zoom

![Shows the declaration with the terms Governments and King of Great Britain highlighted.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/nlu-rich-view.png)

Figure 2. The recognized entities, Governments and King of Great Britain, are highlighted

From the JSON view of the document, you can see the underlying JSON structure of the entity mentions.

Zoom

![Shows the JSON view of the Systems of Government and King of Great Britain entities that are identified in the document](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/nlu-json-view.png)

Figure 3. JSON representation of recognized entity mentions

If you want to search for the Organization entity type, for example, you can copy all of the JSON content into a text editor and search for Organization. Click the Copy icon from the root of the JSON tree view.

Zoom

![Shows the copy icon at the root of the JSON tree](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/nlu-json-root-copy.png)

Figure 4. Copy icon at the root of the JSON representation of enrichments
Non-ideal use cases Write-intensive workloads, random write operations, incremental data updates, or transaction databases. 







 Databases 

Virtual Private Cloud

Classic infrastructure

Satellite



 Characteristics Description 

 Ideal data types Depends on the DBaaS 
 Data usage pattern Read-write-intensive workloads 
 Access Via REST API from your app. 
 Supported Kubernetes access writes N/A as accessed from the app directly. 
 Performance High if deployed to the same data center as your app. 
 Consistency Depends on the DBaaS 
 Durability High 
 Resiliency Depends on the DBaaS and your setup. 
 Availability High if you set up multiple instances. 
 Scalability Scales automatically 
 Encryption At rest 
 Backup and recovery Depends on the DBaaS 
 Common use cases Multizone clusters, relational and non-relational databases, or geographically distributed data. 
 Non-ideal use cases App that is designed to write to a file system.