vania-janet/multiturn-rag-retrieval-data huggingface dataset:
In the Consolidated cluster section, accept or change the default name of the consolidated cluser. For more information, see [Cluster name](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-cr_orderinginstance-consolidwkldcr_orderinginstance-consolidwkld-cluster-name).



1. For the Data center location, select the Geography, Data Center, and Pod to host the clusters. For more information, see [Data center location](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-cr_orderinginstance-consolidwkldcr_orderinginstance-consolidwkld-dc-location).
2. For the CPU model, select either Cascade Lake or SAP-certified Cascade Lake and choose one of the available configurations. For Cascade Lake, also select the RAM size.
3. Select the number of bare metal servers. You can order 4-51 bare metal servers.
4. If you want to use NFS storage, select the corresponding option.



* To add and configure file shares individually, toggle the Configure shares individually switch on. You must select at least one file share. Then, click Add shared storage and select the Size (GB) and Performance for each file share.
* To add and configure the same settings to all file shares, specify the Number of shares and then, select the Size (GB) and Performance for all the shares.



5. If you want to use vSAN storage, select the corresponding option.



1. Select the size for and number of vSAN capacity disks.
2. Select the size for and number of vSAN cache disks.
3. Select the box if you want to Enable vSAN deduplication and compression.
4. For the vSAN license, use the IBM-provided license by selecting Include with purchase and select the license edition.



6. Select the Networking type, either Public and private network or Private network only.
7. Select the Uplink speed.
Planning for VMware vSphere 

Review the following requirements before you order a VMware vSphere® instance. Plan your VMware vSphere based on the IBM Cloud® data center location and your workload capacity requirements.

You are responsible for setting up the environment, installing, and configuring various VMware® components after the VMware ESXi™ servers are deployed. The following examples are VMware components: VMware vCenter Server®, VMware NSX®, and VMware vSAN™.



 Account requirements 

The account that you are using must meet certain requirements. For more information, see [Signing up for required accounts](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-signing_required_accounts).





 IBM Cloud data center availability 

The vSphere deployment has strict requirements on the physical infrastructure. Therefore, you can deploy clusters only in IBM Cloud data centers that meet the requirements. The following IBM Cloud data centers are available for vSphere deployment.

Cascade Lake bare metal servers are available in

multizone regionIBM Cloud data centers. For more information, see [Multizone region (MZR) overview](https://cloud.ibm.com/docs/loadbalancer-service?topic=loadbalancer-service-multi-zone-region-mzr-overview).

If you select a vSAN component, the location list is filtered by SSD (Solid-State Disk) availability.

Asia-Pacific

Europe

NA East



Table 1. Available IBM Cloud data centers for VMware vSphere instances

 Geography Data center Pod Server options<br><br>[1] 

 Asia-Pacific CHE01 01 Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA21 01 Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA22 01 Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA23 01 Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SNG01 02 Skylake, Cascade Lake, SAP-certified Cascade Lake
Cloud High-Speed Transfer Service protects data as it is uploaded to the bucket and [can greatly reduce transfer time](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-object-storage-simplifies-accelerates-data-to-the-cloud).



1. Download the [City of Los Angeles / Traffic Collision Data from 2010](https://data.lacity.org/api/views/d5tf-ez2w/rows.csv?accessType=DOWNLOAD) CSV file. The file is 81MB and may take a few minutes to download.
2. In your browser, access the data-lake-cos service instance from the [Resource List](https://cloud.ibm.com/resources) under the Storage section.
3. Create a new bucket to store data.



* Click Create a bucket.
* Select Custom bucket/Customize your bucket.
* Close to the top of the form, provide a bucket Name.
* Select Regional in the Resiliency section.
* Select a Location.
* At the bottom of the form click Create bucket.



4. Upload the CSV file to Object Storage.



* From your bucket, click Upload.
* Select Standard transfer to use regular http file transfer or select the Aspera high-speed transfer radio button, you may need to install the Aspera plugin to your machine.
* Click Upload files.
* Browse and select the previously downloaded CSV file and click Upload.









 Step 3: Working with data 

In this section, you will convert the original, raw dataset into a targeted cohort based on time and age attributes. This is helpful to consumers of the data lake who have specific interests or would struggle with very large datasets.

You will use Data Engine to manipulate the data where it resides in Object Storage using familiar SQL statements. Data Engine has built-in support for CSV, JSON and Parquet - no additional computation services or extract-transform-load is necessary.



1.
You can deploy the cluster to a different IBM Cloud data center than the deployed instance if you ensure that the network latency between the two IBM Cloud data centers is less than 150 ms.

If you deploy the cluster to a different IBM Cloud data center or IBM Cloud infrastructure pod, three extra VLANs are ordered for use with the ordered IBM Cloud bare metal servers.





 Skylake 

For Skylake servers, you can choose the following CPU models and a supported RAM size, which depends on the NSX networking solution.

Available options might differ depending on the version that your instance was initially deployed in.

Skylake servers are not supported for VMware vSphere® Enterprise Plus 7.0u1 instances.

NSX-T instances

NSX-V instances



Table 1. Options for Skylake bare metal servers - NSX-T instances

 CPU model Cores GHz RAM 

 Dual Intel® Xeon® Silver 4110 processor 16 2.1 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 5120 processor 28 2.2 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 6140 processor 36 2.3 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 







 Cascade Lake 

For Cascade Lake servers, you can choose the following CPU models and a supported RAM size, which depends on the NSX networking solution.

NSX-T instances

NSX-V instances



Table 2. Options for Cascade Lake bare metal servers - NSX-T instances

 CPU model Cores GHz RAM 

 Dual Intel Xeon Silver 4210 processor 20 2.2 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 5218 processor 32 2.3 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 6248 processor 40 2.5 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 6250 processor 16 3.9 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB
2. Select the size for and number of vSAN cache disks.
3. Select the box if you want to Enable vSAN deduplication and compression.
4. For the vSAN license, use the IBM-provided license by selecting Include with purchase and select the license edition.



6. Select the Networking type, either Public and private network or Private network only.
7. Select the Uplink speed. The 25 Gb option is available only for specific pods and data center locations.
8. Specify the VLAN settings:



* To order new public and private VLANs, choose Order new VLANs. A new public VLAN and two private VLANs are provisioned.
* To reuse existing public and private VLANs, if they are available, choose Select existing VLANs. Specify the VLANs and the subnets.





7. If you want to deploy a separate workload cluster, in the Workload cluser section, toggle the Deploy separate workload cluster switch on and specify the cluster settings:



1. Accept or change the default name of the workload cluster.
2. For the Data center location, select the Geography, Data Center, and Pod to host the clusters. For more information, see [Data center location](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-cr_orderinginstance-consolidwkldcr_orderinginstance-consolidwkld-dc-location).
3. For the CPU model, select either Cascade Lake or SAP-certified Cascade Lake and choose one of the available configurations. For Cascade Lake, also select the RAM size.
4. Select the number of bare metal servers. You can order 4-59 bare metal servers.
5. If you want to use NFS storage, select the corresponding option.



* To add and configure file shares individually, toggle the Configure shares individually switch on. You must select at least one file share. Then, click Add shared storage and select the Size (GB) and Performance for each file share.
Ingesting data from object storage bucket 

In this tutorial, you learn to move data into a data lake or an object storage bucket and load the data files to Presto. You will also learn to optimize the file format to choose the table format and run complex SQL query in IBM® watsonx.data.

Sample Scenario : You need to run SQL query on data files that is in your object storage bucket. For this, you must attach the data files in your object storage bucket to Presto. You can also convert data into an optimized analytical format in Parquet or ORC to enhance query performance and reduce server and storage resource consumption. Now, you can run SQL query against the table you created.



 Objective 



* Creating infrastructure within the watsonx.data service.
* Establishing connection with the customer data bucket.
* Querying from the bucket







 Before you begin 

This tutorial requires:



* Subscription of watsonx.data on cloud.
* The configuration details of data bucket that you bring in. This is required for establishing connection with the watsonx.data.
* Ensure that the data bucket has data.







 Step 1: Uploading data into an object storage bucket and attaching to Presto 

In this section of the tutorial, you are going to manage data in an object storage bucket and attach the bucket to HMS and associate with Presto engine.



1. Access any one of the object storage access tools like S3 Browser, AWS S3 console, direct S3 APIs, and various CLI/UI object storage tools.
2. Load data files to your object storage bucket by using the tool.
3. Register and attach the object storage bucket to HMS and associate with Presto engine by using watsonx.data UI.
4. Alternatively, you can also register and attach an object storage bucket with pre-existing data to HMS.

ORC, Parquet, Avro, RCFile, SequenceFile, JSON, Text (CSV) are the Hive supported file formats.







 Step 2: Load data files into Presto 

After attaching the object storage bucket to HMS, you need to load data files into Presto by creating schema and external tables through the Hive connector.



1.
* Improve data residency compliance by keeping data at-rest within specific locations such as the EU-managed locations. See [EU-managed account scenario](https://cloud.ibm.com/docs/atracker?topic=atracker-atracker-scenarios&interface=cliscenarioss-2).
* Consolidate Activity Tracker Event Routing data to the account and region of your primary operations. You can also route auditing events to multiple locations. See [Enterprise account scenario](https://cloud.ibm.com/docs/atracker?topic=atracker-atracker-scenarios&interface=cliscenarioss-3).
* Manage global events in an account and route them to the location of your choice. See [Global events scenario](https://cloud.ibm.com/docs/atracker?topic=atracker-atracker-scenarios&interface=cliscenarioss-4).
* Route data to a Data Lake for analytics. See [Data lake scenario](https://cloud.ibm.com/docs/atracker?topic=atracker-atracker-scenarios&interface=cliscenarioss-5).
* Route data to





 Financial Services (FS) Validated scenario 

Forward your activity event data to IBM Cloud Object Storage for IBM Cloud for Financial Services compliance.

If you require a solution that is Finantial Services (FS) Validated, you can run your business in IBM Cloud in any of the supported regions such as us-south, us-east, eu-de, and eu-gb.

Therefore, to configure Activity Tracker Event Routing for your account, you must define your account settings, targets and routes as follows:



* You must set your account Metadata region primary setting to any of the supported FS validated regions in IBM Cloud. For example, you can set it to us-east.
* You must set your account Permitted target region to the set of FS Validated regions where you operate. For example, you can set it to: us-south,us-east,eu-gb,eu-de. This setting will prevent administrators from creating targets in non-allowed locations.
Bare metal server 

When you size the capacity of your servers, consider your current requirements, and include extra capacity to accommodate anticipated growth. For more information about sizing properly, see [Exporting VMware inventory](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vmware-inventory-export).



 Data center location 

Select the IBM Cloud® data center settings. For more information, see [Region and data center locations for resource deployment](https://cloud.ibm.com/docs/overview?topic=overview-locations).



 Geography 

Select the region where the cluster is to be hosted.





 Data center 

Select the IBM Cloud data center where the cluster is to be hosted. If you select a VMware vSAN™ component, the location list is filtered by SSD availability.





 Pod 

Select the IBM Cloud data center pod where you want to deploy your resources. Keep the default pod selection if you do not have reasons to prefer a different pod.







 CPU model and RAM 



 Skylake 

For Skylake servers, you can choose the following CPU models and a supported RAM size. Options available depend on whether you selected the vSAN component.

Skylake servers are not supported for VMware vSphere® Enterprise Plus 7.0 instances.



Table 1. Options for Skylake bare metal servers

 CPU model Cores GHz RAM 

 Dual Intel® Xeon® Silver 4110 processor 16 2.1 64 GB, 96 GB, 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 5120 processor 28 2.2 64 GB, 96 GB, 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 
 Dual Intel Xeon Gold 6140 processor 36 2.3 64 GB, 96 GB, 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB 







 Cascade Lake 

For Cascade Lake servers, you can choose the following CPU models and a supported RAM size.



Table 2. Options for Cascade Lake bare metal servers

 CPU model Cores GHz RAM
Planning for vCenter Server instances 

Plan your instance based on the IBM Cloud® data center location, your workload capacity requirements, and add-on services requirements. Review the following requirements before you order your VMware vCenter Server® instance.



* New deployments of vCenter Server instances with VMware vSphere® 6.5 or 6.7 are not supported.
* New deployments of vCenter Server multizone instances are not supported.
* New deployments of vCenter Server with NSX-V instances are not supported.





 Account requirements 

The account that you are using must meet certain requirements. For more information, see [Signing up for required accounts](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-signing_required_accounts).





 IBM Cloud data center availability 

The vCenter Server deployment has strict requirements on the physical infrastructure. Therefore, you can deploy instances only in IBM Cloud data centers that meet the requirements. The following IBM Cloud data centers are available for vCenter Server deployment.

Asia-Pacific

Europe

NA East



Table 1. Available IBM Cloud data centers for vCenter Server instances

 Geography Data center Pod Server options for NSX-T<br><br>[1] Server options for NSX-V<br><br>[2] 

 Asia-Pacific CHE01 01 Skylake, Cascade Lake Skylake, Cascade Lake, SAP-certified Cascade Lake<br><br>[3] 
 Asia-Pacific OSA21 01 Cascade Lake, SAP-certified Cascade Lake Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA22 01 Cascade Lake, SAP-certified Cascade Lake Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific OSA23 01 Cascade Lake, SAP-certified Cascade Lake Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SNG01 02 Skylake, Cascade Lake, SAP-certified Cascade Lake Skylake, Cascade Lake, SAP-certified Cascade Lake 
 Asia-Pacific SYD01 01-02 Skylake, Cascade Lake, SAP-certified Skylake, Cascade Lake, SAP-certified Cascade Lake
For more information about how to retrieve an instance ID, see [Obtaining the service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverlessendpoints-cli).
* hms-thrift-endpoint-from-watsonx.data: Specify the credentials for watsonx.data.
* hms-user-from-watsonx.data: The watsonx.data username.
* hms-password-from-watsonx.data: The watsonx.data password.







 Configuring Analytics Engine instance by using Analytics Engine CLI 

To specify the configuration settings for your IBM Analytics Engine instance from CLI, complete the following steps:

Run the following command :

ibmcloud analytics-engine-v3 instance default-configs-update [--id INSTANCE_ID] --body BODY

Parameter value:



* BODY: Copy and paste the following configuration information:

{
"spark.sql.catalogImplementation": "hive",
"spark.driver.extraClassPath": "/opt/ibm/connectors/iceberg-lakehouse/iceberg-3.3.2-1.2.1-hms-4.0.0-shaded.jar",
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
"spark.sql.iceberg.vectorization.enabled": "false",
"spark.sql.catalog.lakehouse": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.lakehouse.type": "hive",
"spark.sql.catalog.lakehouse.uri": "<hms-thrift-endpoint-from-watsonx.data> for example (thrift://81823aaf-8a88-4bee-a0a1-6e76a42dc833.cfjag3sf0s5o87astjo0.databases.appdomain.cloud:32683) ",
Gateway cluster 

An edge gateway is required. You can bring your own gateway appliance or choose from one of the following.



* Gateway cluster with Juniper® vSRX
* Gateway cluster with FortiGate® Virtual Appliance
* FortiGate Security Appliance



The gateway cluster is deployed in the same data center as the consolidated cluster.

The gateway cluster is available for all firewall appliances except for FortiGate Security Appliance.

The data center of the consolidated cluster must be available for gateway cluster deployment. Gateway cluster deployment is not supported for Dallas 09.



 Cluster name 

By default, the gateway cluster name is set to instance name-edge.

You can also specify a cluster name that meets the following requirements:



* Only lowercase alphabetic, numeric, and dash (-) characters are allowed.
* The cluster name must start with a lowercase alphabetic character.
* The cluster name must end with a lowercase alphabetic or numeric character.
* The maximum length of the cluster name is 10 characters.
* The cluster name must be unique within your account.







 Compute capacity 

For compute capacity, you get two cluster hosts and two 3.8 TB SSD storage.





 CPU model 

For the CPU model, the default is a Cascade Lake Dual Intel Xeon Silver 4210 with 20 cores, 2.20 GHz.

You can also select a Cascade Lake Dual Intel Xeon Gold 5218 with 32 cores, 2.30 GHz.





 RAM 

You can select a RAM size from 64 GB, 96 GB, 128 GB, 192 GB, 384 GB, 768 GB, and 1.5 TB. The default is 768 GB.





 Uplink speed 

The uplink speed provides two options:



* 10 Gb, which is selected by default.
* 25 Gb, which is available only for specific pods and data center locations. The following table shows the available locations for the 25 Gb uplink speed.



Asia-Pacific

Europe

NA East



Table 1. Available locations for 25 Gb uplink speed

 Geography Data center Pod 

 Asia-Pacific TOK02 02 
 Asia-Pacific TOK04 01 
 Asia-Pacific TOK05 01 







 Networking type
* The saved portable subnet settings are cleared if you change the Public VLAN, Private VLAN, or Secondary private VLAN settings.



IBM Cloud for VMware Solutions takes control of the entire subnet and you can't use any IP addresses in the subnet.











 System settings for gateway clusters 

When you add a gateway cluster to a vCenter Server instance, you must specify the following settings.



 Data center location 

Select the IBM Cloud® data center settings. For more information, see [Region and data center locations for resource deployment](https://cloud.ibm.com/docs/overview?topic=overview-locations).



 Geography 

Select the region where your consolidated cluster or instance is hosted.





 Data center 

Select the IBM Cloud data center where the consolidated cluster is hosted.





 Pod 

Select the IBM Cloud data center pod where you want to deploy your resources. Keep the default pod selection if you do not have reasons to prefer a different pod.







 Cluster name 

By default, the cluster names are set to instance name-edge.

You can also specify a new name for your clusters. The names must meet the requirements that are listed in [Cluster name](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_orderinginstance-consold-clustervc_orderinginstance-consold-cluster-name).





 CPU model 

You can choose the following CPU models:



* Dual Intel® Xeon® Silver 4210 processor (Cascade Lake)
* Dual Intel Xeon Gold 5218 processor (Cascade Lake)







 RAM 

You can select different values between 64 GB and 1.5 TB.





 Number of bare metal servers 

The number of servers is set to two and cannot be changed. Both servers have the same configuration.





 Hostname prefix 

The hostname prefix applies to all clusters in the instance. It must meet the following requirements:



* Only lowercase alphabetic, numeric, and dash (-) characters are allowed.
* The hostname prefix must start with a lowercase alphabetic character.
* The hostname prefix must end with a lowercase alphabetic or numeric character.
Procedure to order vCenter Server instances 



1. In the VMware Solutions console, click the VMware vCenter Server card in the Platforms section.
2. On the Create tab, enter the instance name and select a resource group.
3. Specify the instance configuration name. Click Browse configurations to open the Instance configuration manager side panel and choose the configuration.



* If you do not see any list of saved configurations in the side panel and want to create a new configuration, you must save the instance settings first as a new configuration template without placing an order.
* If you want to update a saved configuration or create a new configuration based on a saved one, select the configuration from the side panel, make your changes, and then save.



4. Select the instance type:



* Click Primary instance to deploy a single instance in the environment or to deploy the first instance in a multisite topology.
* Click Secondary instance to connect the instance with an existing (primary) instance in the environment for high availability. Select the primary instance that you want the secondary instance to be connected with, then enter the vCenter Server Administrator password for the primary instance.



5. Use the IBM-provided licenses for VMware components, which are included with purchase. For NSX, specify the license edition.

Bring Your Own License (BYOL) is no longer supported except for migrations or upgrades of existing BYOL clusters. Select I will provide and enter your own license key only if you are performing an upgrade or migration of an existing BYOL cluster.
6. Specify the settings for the consolidated cluster:



1. Specify the cluster name.
2. Specify the data center location. Click the Edit icon ![Edit icon](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/icons/edit-tagging.svg) and select the geography, data center, and pod to host the instance.
3. Specify the CPU model: either Cascade Lake or SAP-certified Cascade Lake and choose one of the available configurations. For Cascade Lake, also select the RAM size.