vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Learn more](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutableimmutable-terminology-indefinite).



Add a legal hold flag or an indefinite retention flag to individual archived files if you need to keep a file longer than the default retention period specified at the bucket level.







 Define the strategy to query archived data 

You can download data locally, and then use your own tools to query the data. When you download data for analysis, you are responsible for ensuring that your users comply with the regulations and compliance requirements that may be required for your organization. For example, if you must comply with GDPR, you need to control that users are following the download guidelines per GDPR rules.

You can use the Data Engine service to query IBM Log Analysis archived files that are stored in a COS bucket in your account. You can run queries from the IBM Cloud UI, or programmatically.

Avoid downloading data locally. Use the Data Engine service to query archived data.

The Data Engine service provides a server-less, no-ETL solution to easily query data stored in Object Storage. Underneath, SQL Query uses Apache Spark SQL as its underlying query engine. You can use the Data Engine to run SQL queries (that is, SELECT statements) to analyze, transform structured and semi-structured data, or clean up rectangular data. You cannot run actions such as CREATE, DELETE, INSERT, and UPDATE.

The Data Engine service can process input data that is read from CSV, JSON, ORC, Parquet, or AVRO files. Archived files from an IBM Cloud Activity Tracker instance contain data in JSON format. When you use the Data Engine service, each query result can be written to a CSV, JSON, ORC, PARQUET, or AVRO file in an Object Storageinstance of your choice.

When you query an IBM Cloud Activity Tracker archive file, you must convert the JSON formatted file into PARQUET format to be able to query the contents successfully.
Open data format table support 

The current generation of IBM® Db2® Warehouse on Cloud plans on AWS supports open data formats as DATALAKE tables, allowing for seamless access to other data within your enterprise for integrated workloads. You can:



*  Browse, explore, and query enterprise data in both Db2 and DATALAKE formats, using either the web-based UI, or through SQL
*  Access data in place within DATALAKE tables, joining as necessary with Db2 based data for queries
*  Access data within DATALAKE tables and import into Db2 formatted tables
*  Create new DATALAKE tables in S3 and export from Db2 formatted tables



Supported open table and data formats include Iceberg, Parquet, ORC, and CSV. Both regular and Iceberg DATALAKE table types are supported, based on existing data formats or for business/technical requirements such as ACID compliance. For more information about open data format support in Db2 Warehouse on Cloud, see [Using DATALAKE tables](https://www.ibm.com/docs/en/db2woc?topic=using-datalake-tables)

IBM® Db2® Warehouse on Cloud also provides seamless integration and common metadata service with the new IBM watsonx.data service. You can use the web-based UI to register your watsonx.data service, to allow the import and export of data between the services. For more information about integration with watsonx.data, see [watsonx.data Integration](https://www.ibm.com/docs/en/db2woc?topic=watsonxdata-integration)
Stream landing from Event Streams to Cloud Object Storage 

IBM® Event Streams for IBM Cloud® is directly integrated with the IBM Cloud® Data Engine service, allowing for Streams landing from the Event Streams service into IBM Cloud® Data Lake on IBM Cloud® Object Storage. The [Stream landing tutorial](https://www.ibm.com/cloud/blog/stream-landing-from-event-streams-kafka-service-to-ibm-cloud-data-lake-on-object-storage) is a getting started guide that shows you how to run a fully managed stream data ingestion from Kafka into Parquet on Cloud Object Storage.
<-- </ul> -->

<-- </ul> -->

<-- </ul> -->

Sample JSON data: {
"name": "www.example.com",
"fallback_pool": "17b5962d775c646f3f9725cbc7a53df4",
"default_pools":
"17b5962d775c646f3f9725cbc7a53df4",
"9290f38c5d07c2e2f4df57b1f61d4196"
],
"description": "Example global load balancer.",
"ttl": 60,
"region_pools": {
"WNAM":
"de90f38ced07c2e2f4df50b1f61d4194",
"9290f38c5d07c2e2f4df57b1f61d4196"
],
"ENAM":
"00920f38ce07c2e2f4df50b1f61d4194"
] ! !
}
}
Show more
-s, --json-str: Deprecated. The JSON data describing a global load balancer.-j, --json-file: Deprecated. A file contains input JSON data.-i, --instance: Instance name or ID.
BI.S3.H2.192 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2019)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2256)<br><br><br> 
 BI.S3.H2.384 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2020)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2255)<br><br><br> 
 BI.S3.H2.768 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2021)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2254)<br><br><br>
Non-ideal use cases Write-intensive workloads, random write operations, incremental data updates, or transaction databases. 







 Databases 

Virtual Private Cloud

Classic infrastructure

Satellite



 Characteristics Description 

 Ideal data types Depends on the DBaaS 
 Data usage pattern Read-write-intensive workloads 
 Access Via REST API from your app. 
 Supported Kubernetes access writes N/A as accessed from the app directly. 
 Performance High if deployed to the same data center as your app. 
 Consistency Depends on the DBaaS 
 Durability High 
 Resiliency Depends on the DBaaS and your setup. 
 Availability High if you set up multiple instances. 
 Scalability Scales automatically 
 Encryption At rest 
 Backup and recovery Depends on the DBaaS 
 Common use cases Multizone clusters, relational and non-relational databases, or geographically distributed data. 
 Non-ideal use cases App that is designed to write to a file system. 







 OpenShift Data Foundation 

Virtual Private Cloud

Classic infrastructure

Satellite



 Characteristic Description 

 Supported infrastructure providers Classic, VPC, Satellite 
 Ideal data types Any 
 Data usage pattern Write-intensive workloads. Random read and write operation. Sequential read and write operations. 
 Access Via file system on mounted volume or via REST API from your app. 
 Supported write access levels All 
 Performance Close to bare metal performance for sequential read and write operations when you use SDS machines. Create a storage layer based on the storage class performance (IOPs) that you need. 
 Consistency Strong 
 Durability Very high as three copies of your data are always maintained. 
 Resiliency High when set up with replication across three zones. Medium, when you store data in a single zone only. 
 Availability High when you replicate data across three worker nodes in different zones. 
 Scalability Increase volume capacity by resizing the volume. To increase overall storage layer capacity, you must add worker nodes or remote block storage. Both scenarios require monitoring of capacity by the user. 
 Encryption Bring your own key with IBM Key Protect or HPCS. In-transit and at rest. 
 Backup and recovery Use local or cloud snapshots to save the current state of a volume.
requests(filter:...) {
date, time, bytes,...
}
}
}

The initial node of the user running the query is viewer. A viewer is able to access one or more domains (zones). Each zone contains different datasets, such as firewall events for a zone.

Nodes that represent aggregated data include the groups suffix, for example, firewallEventsAdaptiveGroups. Each group follows a specific structure, shown in the following example.

type ExampleGroup {
count No subfields, it is just the group size.
sum {
 fields that support summing (numbers, maps of numbers)
}
avg {
 fields that support averaging (numbers)
}
uniq {
 fields that support uniqueing (numbers, strings, enums, IPs, dates, etc.)
}
}

This example shows a valid group:

httpRequests1mGroups {
sum {
bytes
}
uniq {
uniques unique IPs
}
dimensions {
datetimeMinute
}
}





 Datasets 

The following datasets are available.



Table 1. Available datasets

 Dataset Node 

 Browser Insights browserInsightsAdaptiveGroups 
 Firewall Activity Log firewallEventsAdaptivefirewallEventsAdaptiveByTimeGroups 
 Firewall Analytics firewallEventsAdaptiveGroups 
 Health Check Analytics healthCheckEventshealthCheckEventsGroups 
 HTTP Requests httpRequests1mGroupshttpRequests1hGroupshttpRequests1dGroupshttpRequestsAdaptiveGroups 
 Image Resizing Analytics imageResizingRequests1mGroups 
 Load Balancing Analytics loadBalancingRequestsloadBalancingRequestsGroups 
 SYN Attacks (DoS Analytics) synAvgPps1mGroups 
 Edge Functions Metrics workersInvocationsAdaptive 







 Errors
A JavaScript object that represents a back-end service such as a database, REST API (to be consumed), or SOAP web service. Data sources are backed by connectors that then communicate directly with the database or other back-end services.





 LoopBack model 

A model that provides a remote (REST) API that clients use to perform operations and interact with backend systems. The model consists of application data, validation rules, data access capabilities, and business logic. Every LoopBack application by default has a set of built-in models: user, application, email, and several models for access control.





 LUN 

See [logical unit number](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2163327).







 M 



 machine learning (ML) 

A method of data analysis that iteratively learns from past data and independently adapts when exposed to new data. The mathematical model at the core of machine learning is built from ground truth inputs. Through training and refinement of example input data, the model can deliver accurate, repeatable results when it analyzes new data.





 machine learning annotator 

See [machine learning model](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx7579194).





 machine learning model 

A component that identifies entities and entity relationships according to a statistical model that is based on ground truth. The model applies past experience, such as training data, to determine or predict the correct outcome of future experiences based on characteristics of the data. These past experiences are captured in the form of a model by calculating feature scores for each candidate answer or evidence and combining that with known outcomes.





 Managed Service Provider (MSP) 

An IBM Business Partner that provides IT services on a contractual basis to maintain clients' computers, networks or software. They manage services on-site at the clients' data center, remotely in the clients’ data center, or in a third-party data center.





 master key 

An encryption key that is used to protect a crypto unit. The master key provides full control of the hardware security module and ownership of the root of trust that encrypts the chain keys, including the root key and standard key.
,"name":"www.example.com","ttl":60,"fallback_pool":"17b5962d775c646f3f9725cbc7a53df4","default_pools":"17b5962d775c646f3f9725cbc7a53df4","9290f38c5d07c2e2f4df57b1f61d4196"],"region_pools":{"WNAM":"de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"ENAM":"00920f38ce07c2e2f4df50b1f61d4194"] ! ! ! }}' -i "cis-demo"
<-- </section "id="section-update-glb-examples" "> --><-- </section "id="section-update-glb" "> --><-- <section "id="section-show-glb" "> --> ibmcloud cis glb Show a global load balancer under a given DNS domain. ibmcloud cis glb DNS_DOMAIN_ID GLB_ID -i, --instance INSTANCE] --output FORMAT] ! ! ! ! ! !
<-- <section "id="section-show-glb-options" "> --> Command options DNS_DOMAIN_ID: The ID of DNS domain. Required.GLB_ID: The ID of global load balancer. Required.-i, --instance: Instance name or ID.
<-- </ul> -->

<-- </ul> -->

<-- </ul> -->

Sample JSON data: {
"name": "www.example.com",
"fallback_pool": "17b5962d775c646f3f9725cbc7a53df4",
"default_pools":
"17b5962d775c646f3f9725cbc7a53df4",
"9290f38c5d07c2e2f4df57b1f61d4196"
],
"description": "Example global load balancer.",
"ttl": 60,
"region_pools": {
"WNAM":
"de90f38ced07c2e2f4df50b1f61d4194",
"9290f38c5d07c2e2f4df57b1f61d4196"
],
"ENAM":
"00920f38ce07c2e2f4df50b1f61d4194"
] ! !
}
}
Show more
-s, --json-str: Deprecated. The JSON data describing a global load balancer.-j, --json-file: Deprecated. A file contains input JSON data.-i, --instance: Instance name or ID. If not set, the context instance specified by ibmcloud cis instance-set INSTANCE is used.--output: Specify output format, only JSON is supported.<-- </section "id="section-create-glb-options" "> --><-- <section "id="section-create-glb-examples" "> --> Examples Create a global load balancer in domain 31984fea73a15b45779fa0df4ef62f9b.
BI.S4.H8.6000 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2303)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2305)<br><br><br> 
 mx2-8x64 IBM Cloud Virtual Server <br><br> * [Business One Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2563)<br><br><br> 
 mx2-16x128 IBM Cloud Virtual Server <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2369)<br> * [Business One Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2418)<br><br><br> 
 mx2-32x256 IBM Cloud Virtual Server <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2370)<br> * [Business One Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2419)<br><br><br>
Non-ideal use cases Write-intensive workloads, random write operations, incremental data updates, or transaction databases. 







 Portworx 

Virtual Private Cloud

Classic infrastructure



 Characteristics SDS (Portworx) 

 Ideal data types Semi-structured and unstructured data 
 Data usage pattern Read-intensive workloads. Few or no write operations. 
 Access Via file system on mounted volume (plug-in) or via REST API from your app 
 Supported Kubernetes access writes <br><br> * ReadWriteMany (RWX)<br> * ReadOnlyMany (ROX)<br> * ReadWriteOnce (RWO)<br><br><br> 
 Performance High for read operations. Predictable due to assigned IOPS and size when you use non-SDS machines. 
 Consistency Eventual 
 Durability Very high as data slices are dispersed across a cluster of storage nodes. Every node stores only a part of the data. 
 Resiliency High as data slices are dispersed across three zones or regions. Medium, when set up in a single zone only. 
 Availability High due to the distribution across zones or regions. 
 Scalability Scales automatically 
 Encryption In transit and at rest 
 Backup and recovery Data is automatically replicated across multiple nodes for high durability. For more information, see the SLA in the [IBM Cloud Object Storage service terms](http://www.ibm.com/support/customer/csol/terms/?id=i126-7857&lc=en). 
 Common use cases Multizone clusters. Geographically distributed data. Static big data. Static multimedia content 
 Non-ideal use cases Write-intensive workloads, random write operations, incremental data updates, or transaction databases. 







 Databases 

Virtual Private Cloud

Classic infrastructure

Satellite



 Characteristics Description 

 Ideal data types Depends on the DBaaS 
 Data usage pattern Read-write-intensive workloads 
 Access Via REST API from your app. 
 Supported Kubernetes access writes N/A as accessed from the app directly. 
 Performance High if deployed to the same data center as your app. 
 Consistency Depends on the DBaaS
-D, --domain
: Filter by domain portion of the FQDN.

-d, --datacenter
: Filter by datacenter shortname.

-H, --hostname
: Filter by host portion of the FQDN.

-m, --memory
: Filter by memory in megabytes.

-n, --network
: Filter by network port speed in Mbps.

-P, --public-ip
: Filter by public IP address.

-p, --private-ip
: Filter by private IP address.

--hourly
: Show only hourly instances.

--monthly
: Show only monthly instances.

-g, --tag
: Filter by tags. This option can be specified multiple times.

-o, --order
: Filter by ID of the order that purchased this instance.

--owner
: Filtered by ID of user who owns the instances.

--sortby
: Column to sort by, default is: hostname, options are: id, hostname, domain, datacenter, cpu, memory, public_ip, private_ip.

--column
: Column to display. Options are: id, hostname, domain, cpu, memory, public_ip, private_ip, datacenter, action, guid, power_state, created_by,tags. This option can be specified multiple times.

Examples:

ibmcloud sl vs list --domain ibm.com --hourly --sortby memory

This command lists all hourly-billing virtual server instances on current account filtering domain equals to "ibm.com" and sort them by memory.





 ibmcloud sl vs options 

List options for creating virtual server instance:

ibmcloud sl vs options [OPTIONS]

Examples:

ibmcloud sl vs options

This command lists all the options for creating a virtual server instance, eg.datacenters, cpu, memory, os, disk, network speed, etc.





 ibmcloud sl vs pause 

Pause an active virtual server instance:

ibmcloud sl vs pause IDENTIFIER [OPTIONS]
Data layout and modeling 

Planning your data model and distribution of your data in IBM Cloud® Object Storage can lead to increased performance and lower costs. The article [How to lay out Big Data in IBM Cloud Object Storage for Spark SQL](https://www.ibm.com/cloud/blog/big-data-layout) shares tricks of the trade to impact cost and performance of your SQL queries.

In Data Engine you can specify the format and layout in which a result for an SQL query is written. By adding these abilities, IBM Cloud® Data Engine opens up serverless data transformation in Cloud Object Storage. For more information, see [Data Engine Releases Serverless Transformation and Partitioning of Data in Open Formats](https://www.ibm.com/cloud/blog/announcements/sql-query-releases-serverless-transformation-and-partitioning-of-data-in-open-formats).
BI.S4.H2.1500 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2253)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2191)<br><br><br> 
 BI.S4.H2.3000 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2192)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2251)<br> * [OLAP Scale-Out Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2425)<br><br><br> 
 BI.S4.H4.3000 (VMware) IBM Cloud for VMware (Local SSD) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2212)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2248)<br><br><br>
Create an instance of [Object Storage](https://cloud.ibm.com/catalog/services/cloud-object-storage). If you already have Object Storage instance with a lite plan, use standard instead of lite.

ibmcloud resource service-instance-create data-lake-cos cloud-object-storage lite global
4. Create an instance of [Data Engine](https://cloud.ibm.com/catalog/services/sql-query). Replace us-south by your region, if needed. If you already have Data Engine instance with a lite plan, use standard instead of lite.

ibmcloud resource service-instance-create data-lake-sql sql-query lite us-south
5. Create an instance of [Watson Studio](https://cloud.ibm.com/catalog/services/watson-studio).

ibmcloud resource service-instance-create data-lake-studio data-science-experience free-v1 us-south
6. Create an API key for service access. Copy the output to a secure, permanent location for later use.

ibmcloud iam api-key-create data-lake-cos-key --output json







 Step 2: Uploading data 

In this section, you will upload data to an Object Storage bucket. You can do this using regular http upload or by utilising the built-in Cloud High-Speed Transfer Service. Cloud High-Speed Transfer Service protects data as it is uploaded to the bucket and [can greatly reduce transfer time](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-object-storage-simplifies-accelerates-data-to-the-cloud).



1. Download the [City of Los Angeles / Traffic Collision Data from 2010](https://data.lacity.org/api/views/d5tf-ez2w/rows.csv?accessType=DOWNLOAD) CSV file. The file is 81MB and may take a few minutes to download.
2.
* [AWS EBS](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refebs-ref)
* [AWS EFS](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refefs-ref)
* [Azure Disk](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refazure-disk-ref)
* [Azure File](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refazure-file-ref)
* [Google Compute Engine](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refgoogle-csi-ref)
* [Local block storage](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-reflocal-block-ref)
* [Local file storage](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-reflocal-file-ref)
* [NetApp ONTAP-NAS 21.04](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refnetapp-nas-ref-2104)
* [NetApp ONTAP-NAS 20.07](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refnetapp-nas-ref)
* [NetApp ONTAP-SAN 21.04](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refnetapp-san-ref-2104)
* [NetApp ONTAP-SAN 20.07](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-class-refnetapp-san-ref)