vania-janet/multiturn-rag-retrieval-data huggingface dataset:
!
<-- <section "id="section-http-request-analytics-options" "> --> Command options DNS_DOMAIN_ID: The ID of DNS domain. Required.--dataset: Requested dataset. The default value is httpRequests1dGroups. Use the following table to identify which datasets are included in your plan and the range of historical data you can query.
<-- <table> -->

Identify datasets included in your plan

 Dataset Trial / Standard Enterprise / Security / GLB 

 httpRequests1dGroups 365 days 365 days 
 httpRequests1hGroups 30 days 90 days 
 httpRequests1mGroups 3 days 7 days 

<-- </table ""> -->

--filter: Filter events. The default value is the last 7 days data. The following operators are supported for all filter options:
<-- <table> -->

Operators supported for filter options

 Operator Comparison 

 gt greater than 
 lt less than 
 geq greater or equal to 
 leq less or equal to 
 neq not equal 
 in in 

<-- </table ""> -->

<-- <ul> -->

* httpRequests1dGroups and httpRequests1hGroups filter options.

<-- <ul> -->

* date

<-- </ul> -->

* httpRequests1mGroups filter options.

<-- <ul> -->

* datetime
* datetimeFifteenMinutes
* datetimeHour
* datetimeDay

<-- </ul> -->

<-- </ul> -->

--order: Output order. (default: "datetime_ASC") The following list is usable order options for corresponding dataset and all of order options support ASC and DESC action. Combine these order options and action with _. For example date_ASC is order by date ascending.
Required.--json: The JSON file or JSON string used to describe a GLB pool.
<-- <ul> -->

* The required fields in JSON data are name, origins, check_regions:

<-- <ul> -->

* name:a short name (tag) for the pool.
* origins:a list of origins within this pool.
* check_regions:a list of geographic region code.

<-- </ul> -->

* The optional fields are description, minimum_origins, enabled, monitor, notification_email.

<-- </ul> -->

Sample JSON data: {
"name": "us-pool",
"description": "application server pool in US",
"origins":
{
"name": "us-app-dal01",
"address": "1.1.1.1",
"enabled": true,
"header": {
"host": "example.com"]
}
},
{
"name": "us-app-dal02",
"address": "2.2.2.2",
"enabled": true
}
],
"minimum_origins": 1,
"check_regions": "WNAM" ] ! ! ,
"monitor": "f1aba936b94213e5b8dca0c0dbf1f9cc",
"enabled": true,
"notification_email": "someone@example.com"
}
Show more
--enable-origin: Enable the origin within the Pool. The value can be ORIGIN_NAME or ORIGIN_ADDRESS.--disable-origin: Disable the origin within the Pool. The value can be ORIGIN_NAME or ORIGIN_ADDRESS.--add-origin: Add an origin into the Pool. ORIGIN_NAME and ORIGIN_ADDRESS are required.
Log analysis 

Learn how to use the Aspera high-speed upload feature of IBM Cloud® Object Storage and how to analyze statistical data in CSV format with [Build a data lake by using Object Storage](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-smart-data-lake).

Learn how to revive your archives to work on log data from IBM® Log Analysis by using IBM Cloud® Data Engine with [Analyze LogDNA log data on IBM Cloud Object Storage by using IBM Cloud Data Engine](https://www.ibm.com/cloud/blog/analyze-logdna-log-data-on-ibm-cloud-object-storage-using-ibm-cloud-sql-query).
"name": "us-pool",
"description": "application server pool in US",
"origins":
{
"name": "us-app-dal01",
"address": "1.1.1.1",
"enabled": true,
"header": {
"host": "test.com"]
}
},
{
"name": "us-app-dal02",
"address": "2.2.2.2",
"enabled": true,
"header": {
"host": "example.com"]
}
}
],
"minimum_origins": 1,
"check_regions": "WNAM" ] ! ! ,
"monitor": "f1aba936b94213e5b8dca0c0dbf1f9cc",
"enabled": true,
"notification_email": "someone@example.com"
}
Show more
-s, --json-str: Deprecated. The JSON data used to describe a GLB pool.-j, --json-file: Deprecated. A file contains input JSON data.-i, --instance: Instance name or ID. If not set, the context instance specified by ibmcloud cis instance-set INSTANCE is used.--output: Specify output format, only JSON is supported.<-- </section "id="section-create-glb-pool-options" "> --><-- <section "id="section-create-glb-pool-examples" "> --> Examples Create a GLB pool for instance cis-demo.
DNS Services Hosted Globally Hosted Globally Hosted Globally Hosted Globally 
 Data Engine (previously SQL Query) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/checkmark-icon.svg) 
 Data Replication Hosted Globally Hosted Globally Hosted Globally Hosted Globally 
 DataStage ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/checkmark-icon.svg) 
 Databases for DataStax ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/icons/checkmark-icon.svg) 
 Databases for EDB !
The Data Engine service can process input data that is read from CSV, JSON, ORC, Parquet, or AVRO files. Archived files from an IBM Cloud Activity Tracker instance contain data in JSON format. When you use the Data Engine service, each query result can be written to a CSV, JSON, ORC, PARQUET, or AVRO file in an Object Storageinstance of your choice.

When you query an IBM Cloud Activity Tracker archive file, you must convert the JSON formatted file into PARQUET format to be able to query the contents successfully.

Use the Data Engine user interface (UI) to develop and test your queries, and the [SQL Query REST API](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-adoptionrestapi) to automate them.

If you plan to use the Data Engine service, and you require HIPAA compliance, create a bucket for your archives that uses a custom key to encrypt the data.
See also [image](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2024928), [container](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2010901).





 LBaaS 

See [load balancer as a service](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx9829528).





 LDAP 

See [Lightweight Directory Access Protocol](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2028538).





 lemma 

The normalized or canonical form of a word. Typically, the lemma is the underived and uninflected form of a noun or a verb. For example, the lemma of the terms 'organizing' and 'organized' is 'organize' See also [surface form](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx3271760), [dictionary](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2001532).





 light API call 

A client operation that only reads data. Light API calls use fewer resources than heavy API calls because they are performing a single function. See also [heavy API call](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx7690468).





 Lightweight Directory Access Protocol (LDAP) 

An open protocol that uses TCP/IP to provide access to directories that support an X.500 model and that does not incur the resource requirements of the more complex X.500 Directory Access Protocol (DAP). For example, LDAP can be used to locate people, organizations, and other resources in an Internet or intranet directory.





 Linux Hyperledger project 

An open source, collaborative effort to advance blockchain technology by identifying and addressing important features for a cross-industry open standard for distributed ledgers that can transform the way business transactions are conducted globally.
In this section, you will convert the original, raw dataset into a targeted cohort based on time and age attributes. This is helpful to consumers of the data lake who have specific interests or would struggle with very large datasets.

You will use Data Engine to manipulate the data where it resides in Object Storage using familiar SQL statements. Data Engine has built-in support for CSV, JSON and Parquet - no additional computation services or extract-transform-load is necessary.



1. Access the data-lake-sql Data Engine service instance from the [Resource List](https://cloud.ibm.com/resources) under the Databases section.
2. Click Launch Data Engine UI under Manage.
3. Create a new dataset by executing SQL directly on the previously uploaded CSV file.



* Replace <your-bucket-name in the URL of theFROM clause with your bucket's name.
* Replace us-south in the URL with the bucket Location (see Configuration tab for the bucket).
* Enter the following SQL into the Type SQL here ... text area.

SELECT
Dr Number AS id,
Date Occurred AS date,
Time Occurred AS time,
Area Name AS area,
Victim Age AS age,
Victim Sex AS sex,
Location AS location
FROM cos://us-south/<your-bucket-name>/Traffic_Collision_Data_from_2010_to_Present.csv
WHERE
Time Occurred >= 1700 AND
Time Occurred <= 2000 AND
Victim Age >= 20 AND
Victim Age <= 35
* Click Run.



4. The Target location will auto-create a Object Storage bucket to hold the result.
5. The intermediate dataset is displayed below the query on the Results tab associated with the Job







 Step 4: Combine Jupyter Notebooks with Data Engine 

In this section, you will use the Data Engine client within a Jupyter Notebook. This re-uses the data stored on Object Storage in a data analysis tool. The combination also creates datasets that are automatically stored in Object Storage that can then be accessed by applications and tools serving line of business users.
BI.S4.H4.3000 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2212)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2248)<br><br><br> 
 BI.S4.H4.6000 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2247)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2213)<br><br><br> 
 BI.S4.H8.6000 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2303)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2305)<br><br><br>
The data model allows for a wide variety of operations ranging across different forms of segmentation or windowing of time series, transformations or conversions of one time series to another, reducers that compute a static value from a time series, joins that join multiple time series, and collectors of time series from different time zones. The time series library enables the plug-and-play of new functions while keeping the core data structure unchangeable. The library also support numeric and categorical typed timeseries.

With time zones and various human readable time formats, a key aspect of the data model is support for Time Reference System (TRS). Every time series is associated with a TRS (system default), which can be remapped to any specific choice of the user at any time, enabling easy transformation of a specific time series or a segment of a time series. See [Using time reference system](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-time-reference-system).

Further, with the need for handling large scale time series, the library offers a lazy evaluation construct by providing a mechanism for identifying the maximal narrow temporal dependency. This construct is very similar to that of a Spark computation graph, which also loads data into memory on as needed basis and realizes the computations only when needed.





 Time series data types 

You can use multiple data types as an element of a time series, spanning numeric, categorical, array, and dictionary data structures.

The following data types are supported in a time series:



Table 1. Time series data types

 Data type Description 

 numeric Time series with univariate observations of numeric type including double and integer. For example:[(1, 7.2), (3, 4.5), (5, 4.5), (5, 4.6), (5, 7.1), (7, 3.9), (9, 1.1)] 
 numeric array Time series with multivariate observations of numeric type, including double array and integer array.
networks = {
0 = {
name = "db-network-1"
ip_allocation_mode = "POOL"
is_primary = true
ip = ""
},
1 = {
name = "isolated-network-1"
ip_allocation_mode = "POOL"
is_primary = false
ip = ""
},
}
disks = {
0 = {
name = "dbDisk"
size_in_mb = "100"
bus_type = "SCSI"
bus_sub_type = "VirtualSCSI"
bus_number = 1
storage_profile = ""
},
1 = {
name = "dbLogDisk"
size_in_mb = "100"
bus_type = "SCSI"
bus_sub_type = "VirtualSCSI"
bus_number = 1
storage_profile = ""
},
}
},
jump-server-1 = {
image = {
catalog_name = "Public Catalog"
template_name = "Windows-2022-Template-Official"
}
memory = 8192
cpus = 2
cpu_hot_add_enabled = true
memory_hot_add_enabled = true
storage_profile = ""
networks = {
0 = {
name = "application-network-1"
ip_allocation_mode = "POOL"
is_primary = true
ip = ""
},
},
disks = {}
},
}
Show more
6. Define public IP address map.

Each virtual data center gets 6 public IP addresses for each virtual data center and its edge gateway. This Terraform template treats the provided consecutive list of IP addresses as a map. The following variable public_ips describes the public IP addresses provided for your virtual data center.
Querying data from data lakes 



 Before you begin 

In the examples, the publicly available [New York taxi trip record data](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) for yellow taxis in January 2022 is used. To follow this example, make sure that the data is in an accessible S3 bucket.





 1. Create an external data source. 

External datasources allow an administrator to grant access to S3 without providing the keys directly to a user.

a) Set ENABLE_EXTERNAL_DATASOURCE.

set ENABLE_EXTERNAL_DATASOURCE = 1;

b) Create an external data source.

create EXTERNAL DATASOURCE 'DATA SOURCE' on 'REMOTE SOURCE'
using (
ACCESSKEYID 'ACCESS KEY ID' SECRETACCESSKEY 'SECRET ACCESS KEY' BUCKET 'BUCKET' REGION 'REGION'
);

Example:

create EXTERNAL DATASOURCE EXAMPLEDATALAKE on AWSS3 
using (
ACCESSKEYID 'XXXX' SECRETACCESSKEY 'XXXX' BUCKET 'exampledatalakebucket' REGION 'US-EAST-1'
);

For more information, see [CREATE EXTERNAL DATASOURCE command](https://www.ibm.com/docs/en/netezza?topic=tables-create-external-datasource-command).





 2. Create an external table. 

After you created an external data source, you can create an external table that accesses the yellow taxi data from January 2022.

Ensure that you have the necessary privileges as described in [Privileges for creating external tables](https://www.ibm.com/docs/en/netezza?topic=et-create-external-table-command-2).

create EXTERNAL table 'TABLE NAME' on 'DATA SOURCE'
using ( 
  DATAOBJECT ('DATA OBJECT') FORMAT 'PARQUET' 
);

The DATAOBJECT argument must reference a single file in the parquet format. If you want to query from multiple parquet files, you must create additional external tables.

Example:
BI.S4.H2.384 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2189)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2250)<br><br><br> 
 BI.S4.H2.768 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2188)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2249)<br><br><br> 
 BI.S4.H2.1500 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2253)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2191)<br><br><br>
* default_pools: A list of pool IDs ordered by their failover priority.
* description: The descriptive text of the load balancer.
* ttl: Time to live (TTL) of the DNS entry for the IP address returned by this load balancer.
* az_pools: A mapping of region and country codes to a list of pool IDs (ordered by their failover priority) for the region.
* enabled: Whether the load balancer pool is enabled. Valid values: true, false. The values on, off will be deprecated.





Sample JSON data:

{
"name": "glb01",
"description": "Global load balancer 01",
"enabled": false,
"ttl": 300,
"default_pools": ["cc1deb57-0eb4-45cb-94ce-a3c71c189d12"],
"fallback_pool": "cc1deb57-0eb4-45cb-94ce-a3c71c189d12",
"az_pools": [
{
"availability_zone": "us-south-1",
"pools":
"0fc0bb7c-2fab-476e-8b9b-40fa14bf8e3d"
]
},
{
"availability_zone": "us-south-2",
"pools":
"f46eba57-f08e-4e38-a76c-f3370d457bd2"
]
}
]
}
Show more

--name
: The DNS hostname to associate with your load balancer.

--fallback-pool
: The pool ID to use when all other pools are detected as unhealthy.

--default-pools
: A list of pool IDs ordered by their failover priority.

--description
: The descriptive text of the load balancer.

--ttl
* fallback_pool:the pool ID to use when all other pools are detected as unhealthy.
* default_pools:a list of pool IDs ordered by their failover priority.

<-- </ul> -->

* The optional fields are description, ttl, region_pools, proxied, enabled, session_affinity, session_affinity_ttl, steering_policy:

<-- <ul> -->

* description: the description of your Load Balancer.
* ttl: time to live (TTL) of the DNS entry for the IP address returned by this load balancer.
* region_pools: a mapping of region/country codes to a list of pool IDs (ordered by their failover priority) for the given region.
* proxied: Control whether or not traffic should flow through the security and performance functions on CIS.
* enabled: Whether to enable (the default) this load balancer.
* session_affinity: valid values are cookie, none.
* session_affinity_ttl: Time, in seconds, until this load balancers session affinity cookie expires after being created.Valid value is between [1800, 604800\. Default is 82800.
* steering_policy: Valid values for steering_policy are off, geo, random, dynamic_latency.

<-- <ul> -->

* off: Use default_pools.
* geo: Use region_pools/pop_pools.
* random: Select a pool randomly.
* dynamic_latency: Use round trip time to select the closest pool in default_pools (requires pool health checks).

<-- </ul> -->

<-- </ul> -->

<-- </ul> -->

Sample JSON data: {
"name": "www.example.com",
"fallback_pool": "17b5962d775c646f3f9725cbc7a53df4",
"default_pools":
Data Engine supports all [public and private Object Storage endpoints](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints). To save space, you can use the alias that is shown instead of the full endpoint name.

Aliases to tethering endpoints (specific endpoints within cross region domains, for example, dal-us-geo) are considered legacy. They continue to work until further notice but are planned to be deprecated sometime in the future. To be prepared, update your applications to use the alias of the corresponding cross region endpoint (for example, us-geo).

Data Engine always uses the internal endpoint to interact with Object Storage, even if an external endpoint was specified in the query. The result location for a query always indicates the external endpoint name. When you interact with Data Engine programmatically through the API, you can use the internal endpoint name to read results instead of the external endpoint name that is returned by the API.

The following tables list some examples of currently supported Data Engine endpoints.



Table 1. Cross region endpoints

 Cross region endpoint name Alias 

 s3.us.cloud-object-storage.appdomain.cloud us-geo 
 s3.eu.cloud-object-storage.appdomain.cloud eu-geo 
 s3.ap.cloud-object-storage.appdomain.cloud ap-geo 





Table 2. Regional endpoints

 Regional endpoint name Alias 

 s3.eu-de.cloud-object-storage.appdomain.cloud eu-de 
 s3.eu-gb.cloud-object-storage.appdomain.cloud eu-gb 
 `s3.us-south.cloud-object-storage.appdomain.cloud us-south` 





Table 3. Single data center endpoints

 Single data center endpoint name Alias 

 s3.ams03.cloud-object-storage.appdomain.cloud ams03 
 s3.mon01.cloud-object-storage.appdomain.cloud mon01 
 s3.sng01.cloud-object-storage.appdomain.cloud sng01
BI.S3.H2.768 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2021)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2254)<br><br><br> 
 BI.S4.H2.192 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2190)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2252)<br><br><br> 
 BI.S4.H2.384 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2189)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2250)<br><br><br>
* fallback_pool: The pool ID to use when all other pools are detected as unhealthy.
* default_pools: A list of pool IDs ordered by their failover priority.



* The optional fields are description, ttl, az_pools, enabled:



* description: The descriptive text of the load balancer.
* ttl: Time to live (TTL) of the DNS entry for the IP address returned by this load balancer.
* az_pools: A mapping of region and country codes to a list of pool IDs (ordered by their failover priority) for the region.
* enabled: Whether the load balancer pool is enabled. Valid values: true, false. The values on, off will be deprecated.





Sample JSON data:

{
"name": "glb01",
"description": "Global load balancer 01",
"enabled": false,
"ttl": 300,
"default_pools": ["cc1deb57-0eb4-45cb-94ce-a3c71c189d12"],
"fallback_pool": "cc1deb57-0eb4-45cb-94ce-a3c71c189d12",
"az_pools": [
{
"availability_zone": "us-south-1",
"pools":
"0fc0bb7c-2fab-476e-8b9b-40fa14bf8e3d"
]
},
{
"availability_zone": "us-south-2",
"pools":
"f46eba57-f08e-4e38-a76c-f3370d457bd2"
]
}
]
}
Show more

--name
: The DNS hostname to associate with your load balancer.

--fallback-pool
: The pool ID to use when all other pools are detected as unhealthy.

--default-pools