vania-janet/multiturn-rag-retrieval-data huggingface dataset:
In classic clusters, if you have multiple VLANs for your cluster, multiple subnets on the same VLAN, or a multizone classic cluster, you must enable a [Virtual Router Function (VRF)](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointvrf) for your IBM Cloud infrastructure account so your worker nodes can communicate with each other on the private network. To enable VRF, see [Enabling VRF](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpointvrf). To check whether a VRF is already enabled, use the ibmcloud account show command. If you can't or don't want to enable VRF, enable [VLAN spanning](https://cloud.ibm.com/docs/vlans?topic=vlans-vlan-spanningvlan-spanning). To perform this action, you need the Network > Manage Network VLAN Spanning[infrastructure permission](https://cloud.ibm.com/docs/containers?topic=containers-access-credsinfra_access), or you can request the account owner to enable it. To check whether VLAN spanning is already enabled, use the ibmcloud ks vlan spanning get --region <region>[command](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_vlan_spanning_get).

--private-only
: Use this option to prevent a public VLAN from being created. Required only when you specify the --private-vlan option and don't include the --public-vlan option. If worker nodes are set up with a private VLAN only, you must enable the private cloud service endpoint or configure a gateway appliance. For more information, see [Planning your private cluster and worker node setup](https://cloud.ibm.com/docs/containers?topic=containers-plan_basicsprivate_clusters).

--output json
: Optional: Prints the command output in JSON format.

-q
Zoom

![IBM Cloud for VMware Solutions network topology](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/vmwaresolutions/images/vcd-sample-topology.svg)

Figure 1. IBM Cloud for VMware Solutions network topology



 Creating a routed Organization virtual data center network 

For more information about how to create a routed organization, see [Add a routed organization virtual data center network](https://docs.vmware.com/en/VMware-Cloud-Director/10.1/VMware-Cloud-Director-Tenant-Portal-Guide/GUID-74C4D27F-9E2A-4EB2-BBE1-CDD45C80E270.html).

Review the following notes for creating a routed Organization virtual data center network.



* Virtual data centers can share networks only if they are in the same region. You can choose to set the Shared option when an application within an Organization virtual data center has a reservation or allocation pool set as the allocation model. In this case, it might not have enough room to run more VMs. As a solution, you can create a secondary Organization virtual data center with on-demand and run more VMs on that network on a temporary basis.
* From the Edge Connection page, select the edge that was created when the virtual data center was created.
* Select Distributed for the Interface Type.
* Repeat the procedure to create another Organization virtual data center network. Give the network a different Name, such as App and a different Gateway CIDR, such as 192.168.101.1/24.







 Validating that the network is connected to the edge 

Use the steps to add a static route to validate that the Organization virtual data center networks that you defined earlier are listed as static routes. When you click Create (+), locate the Next Hop field, and confirm that 52.117.132.2 is displayed. This value is the internal interface of the edge gateway.
Verify Enabled is selected and click KEEP.
9. Click Save Changes.







 Adding the NAT rule for VMware as a Service 

Add a NAT rule for translating internal network addresses into the service network IP address space.



1. Log in to the VMware Cloud Director tenant portal.
2. From the main page under Virtual Data Center, click the virtual data center where you want to add the NAT rule.
3. Under the Networking section, click the Edges tab and open the single preconfigured edge.
4. Under the Services section, click the NAT tab and click NEW to add an SNAT rule.
5. Complete the following configuration in the Add NAT Rule window:



1. Enter a name for the NAT rule.
2. Select SNAT in the Interface Type field.
3. In the External IP field, click the information icon to view the available IP addresses and enter the service IP address that you want to use.
4. In the Internal IP field, enter the internal IP address to be translated.



6. Verify the Advanced Settings fields and click Save.







 Adding the firewall rule for VMware Solutions Shared 

Add a firewall rule to allow the traffic from the internal network to the service network.



1. Click the Firewall tab and click + to add a firewall rule.
2. Add the IP addresses or range from your internal network to the Source field.
3. Add 52.117.132.1/24 in the Destination field.
4. Specify Any for the Service field and Accept for the Action field.
5. Click Save changes.



After the previous configuration is completed, you can use the supported IBM Cloud services on the VMs in your virtual data center.

If your vApp or VM is deployed from the IBM templates that are provided in the public catalog, the services are already configured on the VM. To enable the connection, you must complete the previous steps in Adding the NAT rule and Adding the firewall rule.





 Adding the firewall rule for VMware as a Service 

Add a firewall rule to allow the traffic from the internal network to the service network.
Creating a virtual data center in a VMware as a Service with Terraform 

This tutorial may incur costs. Use the [Cost Estimator](https://cloud.ibm.com/estimator/review) to generate a cost estimate based on your projected usage.

This tutorial is to demonstrate the basic steps to operationalize an IBM Cloud® for VMware as a Service – single tenant instance after initial instance provisioning. This tutorial should take about 20-30 minutes to complete and assumes that [IBM Cloud® for VMware as a Service – single tenant instance](https://cloud.ibm.com/docs/vmware-service?topic=vmware-service-tenant-ordering) and [a virtual data center (VDC)](https://cloud.ibm.com/docs/vmware-service?topic=vmware-service-vdc-adding) have already been provisioned. This tutorial uses an example Terraform template, which can be customized and modified for your use case, if needed.



 Objectives 

In this tutorial, you will learn:



* How to create virtual data center (VDC) networks with Terraform,
* How to create virtual machines on your virtual data center networks with Terraform, and
* How to configure network address translation (NAT) and firewall (FW) rules on your virtual data center edge gateway with Terraform.



The following diagram presents an overview of the solution to be deployed.

Zoom

![Architecture](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution66-vmware-as-a-service/vmwaas-example-diagrams-tf-vmwaas-basic.svg)

Figure 1. Architecture diagram of the tutorial



1. Use IBM Cloud Console to create a virtual data center in your single tenant instance. Your instance may have one or more virtual data centers, so you can have a dedicated virtual data center for testing purposes.
The following SELECT statement combines the observations that have identical time-stamps into a single observation whose value is the original values that are separated by an underscore (_) character:

SELECT TS_COMBINE_DUPLICATE_TIMETICKS(ts1a, TS_COMBINER_CONCATENATE("_"))
FROM cos://us/sql/timeseries.parquet STORED AS PARQUET
INTO <your target location> STORED AS PARQUET

Following, see the result of the example query:

[(1,"a"), (3, "b"), (5, "c_d_e"), (7, "f"), (9, "g")]

Consider the following input DoubleTimeSeries, which is stored in a table column with the name ts1b:

[(1,7.0), (3, 8.5), (5, 9.1), (5, 9.3), (5, 9.8), (7, 10.7), (9, 12.2)]

The following SELECT statement combines the observations that have identical time-stamps into a single observation whose value is the average of the original values.

SELECT TS_COMBINE_DUPLICATE_TIMETICKS(ts1b, TS_COMBINER_AVERAGE())
FROM cos://us/sql/timeseries.parquet STORED AS PARQUET
INTO <your target location> STORED AS PARQUET

Following, see the result of the example query:

[(1,7.0), (3, 8.5), (5, 9.8), (7, 10.7), (9, 12.2)]





 Segmentation 

Segmentation functions create, as output, a segmented version of a time series. Consider the following input DoubleTimeSeries, which is stored in a table column with the name ts2:

[(1, 1.0), (3, 2.0), (5, 3.0), (7, 4.0), (9, 5.0)]

The following SELECT statement uses column ts2 and a window size of 2 and a step size of 1 to generate a new DoubleSegmentTimeSeries:
When the Data Engine query UI opens, a COS bucket is automatically generated. This bucket is used by default by the Data Engine service to store the results from your SQL queries.

Wen you run queries, you can specify a custom bucket to store results in. If your query does not specify one, the default one is used.





 Step 3.2. Get information on the file that you want to query in COS 

Complete the following steps:



1. In the IBM Cloud dashboard, click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/icons/icon_hamburger.svg) > Resource list > Storage.
2. Select the Data Engine instance that has the bucket with the archive files.

Contact your IBM Cloud Activity Tracker administrator to get the COS information.
3. Select Buckets.
4. Select the bucket name. You can see the list of archive files in the bucket.
5. Identify the file that you want to query.

Notice that the file name has the name of your IBM Cloud Activity Tracker instance and the date, in UTC format, of the events that are included.

If you get a file of 20 bytes, that file does not have any data.
6. For that file, select SQL URL.

A window opens that shows the URL.
7. Copy the URL.







 Step 3.3. Get information on the COS bucket that is used to store results from queries 

Complete the following steps:



1. In the COS instance UI, select Buckets.
2. Select the bucket name that you plan to use to store the results from queries.
3. For that bucket, select SQL URL.

A window opens that shows the URL.
4. Copy the URL.







 Step 3.4. Transform an archive file to PARQUET format
Anytime a query is run against "ts", it is run in parallel across time series.

select
key,
ts
from table
using time_series_format(key="key", timetick="time_tick", value="amount") in ts

The issue arises if "ts" is large and thus does not fit in the memory of one machine. To mitigate this issue, modify your key as follows.

First, add a date or any granularity of time to your key (to generate a smaller time series).

select
concat(cast(to_date(time_tick) as str), "-", key) as date_and_key
time_tick,
amount
from table
into table2

Second, create your time series with the new date_and_key key.

select
date_and_key,
ts
from table2
using time_series_format(key="date_and_key", timetick="time_tick", value="amount") in ts

The final queries better use parallelism by providing a larger set of keys. In this case, you not only have the set of unique keys, but the dates associated with each key, as well. By doing so, each time series is much smaller, and has a better chance of fitting in memory.





 How to avoid exhausting allocated resources or out of memory errors (OOM)? 

The most common cause for exhausting allocated resources or out of memory errors is the structure of the query or the layout of the data. The following best practices help to avoid such issues:



* Prepare your data first. Transform input formats, such as .csv or .json to a compressed format, such as Parquet.
* Write results into [partitioned result sets](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencepartitionedClause). Start with a double digit partition value. An optimal single object size is about 128 MB.
After you log in with your user ID and password, the IBM Cloud dashboard opens.
2. Click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/icons/icon_hamburger.svg) > Resource list > Services.
3. Select an Data Engine instance.
4. From the Manage tab, select Launch Data Engine UI.



When the Data Engine query UI opens, a COS bucket is automatically generated. This bucket is used by default by the Data Engine service to store the results from your SQL queries.

Wen you run queries, you can specify a custom bucket to store results in. If your query does not specify one, the default one is used.





 Step 3.2. Get information on the file that you want to query in COS 

Complete the following steps:



1. In the IBM Cloud dashboard, click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/icons/icon_hamburger.svg) > Resource list > Storage.
2. Select the Data Engine instance that has the bucket with the archive files.

Contact your Log Analysis administrator to get the COS information.
3. Select Buckets.
4. Select the bucket name. You can see the list of archive files in the bucket.
5. Identify the file that you want to query.

Notice that the file name has the ID of your Log Analysis instance and the date, in UTC format.
6. For that file, copy the Object Data Engine URL.







 Step 3.3. Get information on the COS bucket that is used to store results from queries 

Complete the following steps:



1. In the COS instance UI, select Buckets.
The result of a NULL predicate cannot be unknown. If the value of the expression is null, the result is true. If the value is not null, the result is false. If NOT is specified, the result is reversed.

The IS DISTINCT FROM predicate compares two expressions and evaluates to TRUE if their values are not identical. The result of a DISTINCT predicate cannot be null. If NOT is specified, the result is reversed. The result of a DISTINCT predicate depends on whether either or both of its input expressions are null:



Table 34. NULL Values and DISTINCT predicate value.

 Input expressions IS DISTINCT FROM IS NOT DISTINCT FROM 

 Both inputs are nonnull. Evaluates to TRUE if the inputs are not identical and FALSE if they are. Equivalent to the <> operator. Evaluates to FALSE if the inputs are not identical and TRUE if they are. Equivalent to the = operator. 
 One input is null. Evaluates to TRUE. Evaluates to FALSE. 
 Both inputs are null. Evaluates to FALSE. Evaluates to TRUE. 



The following DISTINCT predicates are logically equivalent to the corresponding search conditions:



Table 35. DISTINCT predicate and search condition.

 DISTINCT predicate Search condition 

 value 1 IS NOT DISTINCT FROM value2 (value1 IS NOT NULL AND value2 IS NOT NULL AND value1 = value 2) OR (value1 IS NULL AND value2 IS NULL) 
 value 1 IS DISTINCT FROM value2 NOT (value1 IS NOT DISTINCT FROM value2) 







 Examples - predicate 



 IS DISTINCT FROM examples 

-- select all rows with distinct values in column A and B
SELECT * FROM (
SELECT
col1 AS a,
col2 AS b
FROM VALUES
(1 , 2),
(2, 2),
(null, 2),
(1, null),
(2, null),
(null, null)
) WHERE a IS DISTINCT FROM b

The result of the example query is shown in the following table.



Table 36. Query result for example: select all rows with distinct values in column A and B.

 A B 

 1 2
Choose any of the following actions to manage IAM policies in the IBM Cloud:



* To grant permissions to a user, see [Assigning access](https://cloud.ibm.com/docs/account?topic=account-assign-access-resourcesassign-new-access).
* To revoke permissions, see [Removing access](https://cloud.ibm.com/docs/account?topic=account-assign-access-resourcesremoving-access-console)).
* To review a user's permissions, see [Reviewing your assigned access](https://cloud.ibm.com/docs/account?topic=account-assign-access-resourcesreview-your-access-console).







 Step 3. Running a query through the Data Engine UI 

In SQL, the term query is just another way of saying SELECT statement.

To run a query, complete the following steps:



 Step 3.1. Launch the Data Engine query UI 



1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

After you log in with your user ID and password, the IBM Cloud dashboard opens.
2. Click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/icons/icon_hamburger.svg) > Resource list > Services.
3. Select an Data Engine instance.
4. From the Manage tab, select Launch Data Engine UI.



When the Data Engine query UI opens, a COS bucket is automatically generated. This bucket is used by default by the Data Engine service to store the results from your SQL queries.

Wen you run queries, you can specify a custom bucket to store results in. If your query does not specify one, the default one is used.





 Step 3.2. Get information on the file that you want to query in COS 

Complete the following steps:



1.
This command runs a crawl with the configuration file crawler.conf.

The path to the configuration file passed in the --config option must be a qualified path. That is, it must be in relative formats, such as config/crawler.conf or ./crawler.conf, or in an absolute path, such as /path/to/config/crawler.conf.





 Search your documents 

Finally, use the GET /v1/environments/{environment_id}/collections/{collection_id}/query method to search your collection of documents. The following example returns all entities that are called IBM:



* Replace {apikey} with your service credentials.
* Replace {url} with your URL.
* Replace {environment_id} with the environment ID for the environment you created in step 1.
* Replace {collection_id} with the collection ID of the collection that you created in step 2.



curl -u "apikey:{apikey}" "{url}/v1/environments/{environment_id}/collections/{collection_id}/query?version=2019-04-30&query-enriched_text.entities.text:IBM"





 Results 

You successfully queried documents in an environment by using the collection you created. Now, you can begin customizing by adding more documents to the collection.
Select the bucket you are using for archiving.
8. Click the Configuration tab.
9. Copy the Bucket Instance CRN.
10. In the Activity Tracker section make sure the selected service instance is the one you are using. Also make sure Data events is set to read & write.







 Step 2. Define a view to filter the events that report the usage of that service ID 

Complete the following steps to define a view:



1. [Launch the Activity Tracker UI](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-launch). The Views section opens.
2. Select the Everything view.

![Views main page](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/archive-view-1.png)
3. In the search bar, enter the following query:

initiator.id:iam-ServiceId-<GUID> action:cloud-object-storage.object.create target.id:<BUCKET-CRN>

Replace <GUID> with the ID value of the service ID that you copied earlier.

Replace <BUCKET-CRN> with the CRN value of the bucket that you found on the the bucket configuration page.
4. Save the view. For example, you can name the view Upload Frankfurt to indicate archive files into the bucket from an IBM Log Analysis instance located in Frankfurt.

The data that is displayed through the view reports write actions of archive files to the archive bucket.







 Step 3. Define an alert to notify the absence of new archive files 

Complete the following steps to define an absence alert that notifies you when archiving is not happening:



1. Select the view name.

!
Consider the following input DoubleTimeSeries, which is stored in a table column with the name ts2:

[(1, 1.0), (3, 2.0), (5, 3.0), (7, 4.0), (9, 5.0)]

The following SELECT statement uses column ts2 and a window size of 2 and a step size of 1 to generate a new DoubleSegmentTimeSeries:

SELECT TS_SEGMENT(ts2, 2, 1)
FROM cos://us/sql/timeseries.parquet STORED AS PARQUET
INTO <your target location> STORED AS PARQUET

Following, see the result of the example query:

[(1, (1, 1.0), (3, 2.0)]), (3, (3, 2.0), (5, 3.0)]), (5, (5, 3.0),(7, 4.0)]), (7, (7, 4.0),(9, 5.0)])]

Some statistical functions operate expressly on segmented data. For example, the following SELECT statement returns a DoubleTimeSeries. This DoubleTimeSeries contains the average of a DoubleSegmentTimeSeries’s values for each segment of the DoubleSegmentTimeSeries.

SELECT TS_SEG_AVG(TS_SEGMENT(ts2, 2, 1))
FROM cos://us/sql/timeseries.parquet STORED AS PARQUET
INTO <your target location> STORED AS PARQUET

Following, see the result of the example query:

[(1, 1.5), (3, 2.5), (5, 3.5), (7, 4.5)]





 Time reference system (TRS) 

If necessary, use a [TRS](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-TRS) to specify the time granularity of and a start time for a time series. Consider the following input DoubleTimeSeries, which is stored in a table column with the name ts3.
SELECT count(distinct(numbackends)) FROM pg_stat_database;

If you need to figure out where the connections are going, you can break down the connections by database.

SELECT datname, numbackends FROM pg_stat_database;

To further investigate connections to a specific database, query pg_stat_activity.

SELECT * FROM pg_stat_activity WHERE datname='ibmclouddb';





 Terminating Connections 

Your admin user has the pg_signal_backend role. If you find connections that need to be reset or closed, the admin user can use both [pg_cancel_backend and pg_terminate_backend](https://www.postgresql.org/docs/current/functions-admin.htmlFUNCTIONS-ADMIN-SIGNAL-TABLE). The pid of a process is found from the pg_stat_activity table.



* pg_cancel_backend Cancels a connection's current query without terminating the connection, and without stopping any other queries that it might be running.

SELECT pg_cancel_backend(pid);
* pg_terminate_backend Stops the entire process and closes the connection.

SELECT pg_terminate_backend(pid);



The admin user does have the power to reset or close the connections for any user on the deployment except superusers. Be careful not to terminate replication connections from the ibm-replication user, as it interferes with the high-availability of your deployment.



 End Connections 

If your deployment reaches the connection limit or you are having trouble connecting to your deployment and suspect that a high number of connections is a problem, you can disconnect (or end) all of the connections to your deployment.

In the UI, on the Settings tab, there is a button to End Connections to your deployment. Use caution, as it disrupts anything that is connected to your deployment.

Zoom

!