vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* [Next steps](https://cloud.ibm.com/docs/containers?topic=containers-storage-cos-tutorial-quotastorage-cos-quota-next)



[VPC: Setting up authorized IP addresses for IBM Cloud Object Storage](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_vpc_ipstorage_cos_vpc_ip)

[Storage class reference](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_referencestorage_cos_reference)



* [Standard](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_referencestandard)
* [Vault](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_referenceVault)
* [Cold](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_referencecold)
* [Flex](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_referenceflex)







 Setting up IBM Cloud VPC Block Storage 

[Setting up Block Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-vpc-blockvpc-block)



* [Quick start for IBM Cloud Block Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-vpc-blockvpc_block_qs)
* [Adding Block Storage for VPC to your apps](https://cloud.ibm.com/docs/containers?topic=containers-vpc-blockvpc-block-add)
* [Using an existing Block Storage for VPC instance](https://cloud.ibm.com/docs/containers?topic=containers-vpc-blockvpc-block-static)
1.0.295 24 Jun 2021 <br><br>1. The ibmcloud sat storage config and ibmcloud sat storage template commands are now generally available.<br>2. Adds the --cluster and --service-cluster-id option to the ibmcloud sat storage assignment ls command to filter output by the ID of a cluster that you created or the ID of a Satellite-enabled IBM Cloud service cluster.<br>3. Adds the --service-cluster-id option to the ibmcloud sat storage assignment create command to deploy storage drivers to a specific Satellite-enabled IBM Cloud service cluster.<br>4. Updates the help text in various languages.<br><br><br> 
 1.0.275 26 May 2021 <br><br>1. Adds the following changes to storage commands: 1. Adds the --cluster option to the ibmcloud sat storage assignment create command to assign to an individual cluster ID instead of a cluster group.<br>2. Adds --location as a required option to the ibmcloud sat storage config create command and an optional option to the ibmcloud sat storage config ls command.<br>3. Changes the --config option to --config-name in the ibmcloud sat storage config sc add command.<br>4. Adds the optional --output json option to the ibmcloud sat config get, ibmcloud sat config ls, and ibmcloud sat config version get commands.<br>5. Increases the maximum size for the file in the --read-config option of the ibmcloud sat config version create command from 1MB to 3MB.<br>6. Fixes the ibmcloud sat group create command to enable cluster group creation.<br>7. The IAM token that is used for your CLI session is now refreshed 5 minutes before expiration to keep the session active.<br>8. Updates the help text in various languages.<br><br><br> 
 1.0.258 26 Apr 2021 <br><br>1.
claim-file-silve Bound pvc-1efef0ba-0c48-11e8-968a-f6612bb731fb 24Gi RWX ibmc-file-silver 83d
2. Review the ReclaimPolicy and billingType for the storage class.

oc describe storageclass <storageclass_name>

If the reclaim policy says Delete, your PV and the physical storage are removed when you remove the PVC. If the reclaim policy says Retain, or if you provisioned your storage without a storage class, then your PV and physical storage are not removed when you remove the PVC. You must remove the PVC, PV, and the physical storage separately.

If your storage is charged monthly, you still get charged for the entire month, even if you remove the storage before the end of the billing cycle.
3. Remove any pods that mount the PVC. List the pods that mount the PVC. If no pod is returned in your CLI output, you don't have a pod that uses the PVC.

oc get pods --all-namespaces -o=jsonpath='{range .items[]}{"n"}{.metadata.name}{":t"}{range .spec.volumes[]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"

Example output

blockdepl-12345-prz7b: claim1-block-bronze
4. Remove the pod that uses the PVC. If the pod is part of a deployment, remove the deployment.

oc delete pod <pod_name>
5. Verify that the pod is removed.

oc get pods
6. Remove the PVC.

oc delete pvc <pvc_name>
7. Review the status of your PV. Use the name of the PV that you retrieved earlier as VOLUME. When you remove the PVC, the PV that is bound to the PVC is released.
Stream landing 

With stream landing, you can stream your data in real time from a topic to a bucket of your choice. IBM Cloud® Data Engine connects to IBM® Event Streams for IBM Cloud® and copies the data to Cloud IBM Cloud® Object Storage in Parquet format. This capability enables efficient analytics on the new objects created.

Zoom

![Stream landing.](https://cloud.ibm.com/docs-content/v1/content/9fbd9f26e0f1e4aaa38411d9a81ad7bc6897624d/sql-query/images/streams_landing_DE.svg)

Figure 1. Stream landing

You can enable a stream landing job on the Event Streams UI by selecting the required resources, such as Cloud Object Storage bucket, IBM® Key Protect instance, and the Data Engine instance by using a tailored wizard. If you want to stop the streaming job, switch to the Event Streams UI. For more information about configuring stream landing in Event Streams, see [Streaming to Cloud Object Storage by using Data Engine](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-streaming_cos_sql).



 Using Data Engine 

You can also configure a stream landing job directly as an SQL query statement, without using the Event Streams UI.

For event consumption, Data Engine reads data from an Event Streams topic with a simple SQL statement, such as the following example:

SELECT * FROM crn:v1:bluemix:public:messagehub:us-south:a/2383fabbd90354d33c1abfdf3a9f35d5:4d03d962-bfa5-4dc6-8148-f2f411cb8987::/jsontopic STORED AS JSON
EMIT cos://us-south/mybucket/events STORED AS PARQUET
Options and parameters supported in ibm-lh tool 

Ingesting data files from S3 or local location into IBM® watsonx.data is done by using two options that are supported in ibm-lh tool.



1. Command line option
2. Configuration file option

a. Global ingest config section

b. Individual ingest config section



Different options and variables that are supported in a command line and configuration file are listed as follows:



1. Command line option



Table 1. Command line options and variables

 Parameter Description Declaration 

 generate-ddl Generate schema for parquet data file ibm-lh data-copy --generate-ddl 
 ingest-config Configuration file for data migration ibm-lh data-copy --ingest-config <INGEST_CONFIGFILE> 
 source-data-files Data file(s) or folder(s) for data migration. File name ending with / is considered a folder. Single or multiple files can be used. Example: <file1_path>,<file2_path>,<folder1_path> ibm-lh data-copy --source-data-files <SOURCE_DATA_FILE> 
 source-s3-creds S3 credentials of data source ibm-lh data-copy --source-s3-creds <SOURCE_S3_CREDS> 
 staging-location Location where CSV files are converted to Parquet are staged. Even parquet files in some circumstances are staged like, Parquet file is local, Parquet files in local folder, a huge Parquet file(local or S3) all of which may not fit in memory, source file is S3 parquet file, and file from S3 parquet folder which may contain special column types such as date or decimal. Example: <s3://lh-target/staging> ibm-lh data-copy --staging-location <STAGING_LOCATION> 
 staging-s3-creds S3 credentials of staging location ibm-lh data-copy --staging-s3-creds <STAGING_S3_CREDS>
[Allowing trial experience](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_hp_intro)Allowing trial experience

{{ shortdesc }}

IBM watsonx.data



* 6 hours
* 2023-07-07



[Quick start watsonx.data console](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-quick_start)Quick start watsonx.data console

When you log in to the IBM® watsonx.data web console for the first time, you are presented with the quick start wizard. In this tutorial, you learn how to use the quick start wizard to configure the core components and get started with watsonx.data in few minutes.

IBM watsonx.data



* 20 minutes
* 2023-07-07



[Ingesting data from object storage bucket](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_ingest_osbckt)Ingesting data from object storage bucket

{{ shortdesc }}

IBM watsonx.data



* 8 hours
* 2023-07-07



[Setting up infrastructure and querying data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_prov_custbckt)Setting up infrastructure and querying data

{{ shortdesc }}

IBM watsonx.data



* 8 hours
* 2023-07-07



[Joining data from different data source by using watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_join_data)Joining data from different data source by using watsonx.data

{{ shortdesc }}

IBM watsonx.data



* 9 hours
* 2023-07-07
To further reduce data movement, pre- and post-processing can also be conducted in the cloud. LSF's Application Center provides a web portal and restful API, which allows jobs to be submitted directly to the cloud cluster and starts remote visualization tools on the job or data in the cloud. The Application Center built in support for visualization of common output formats. It also has a client component that can be used to upload or download data directly from your laptop to the cloud cluster.

The NFS instance that is deployed with the IBM Spectrum LSF cluster in the cloud can be used as a destination with Data Manager configuration. When the data is available on NFS, it is visible to the management and worker nodes of your Spectrum LSF cluster.





 Stand-alone cluster on IBM Cloud 

With a dedicated or stand-alone cluster on the cloud, your data is resident in the cloud. You can use IBM Cloud Object Storage to bring the data into your IBM Cloud account. Object Storage provides a cheap and reliable option to manage your data in IBM Cloud. When the data is available in Object Storage, you can mount the Object Storage bucket onto the management node or copy the data from the Object Storage bucket to the NFS instance to make the data visible to the management and worker nodes of your Spectrum LSF cluster.





 Additional resources 



* For more information on how to configure and handle issues with data management, see [Four Hybrid Cloud Data Management Challenges](https://www.datacenterknowledge.com/cloud/four-hybrid-cloud-data-management-challenges).
* For more information on setting up a Spectrum LSF multicluster configuration, see [IBM Spectrum LSF multicluster capability](https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=lsf-multicluster-capability).
* For more information on setting up a Spectrum LSF data manager configuration, see [IBM Spectrum LSF Data Manager](https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=lsf-data-manager).
See [Options and variables supported in ibm-lh tool](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-cli_commands).

Following are the details of the command line option to ingest data files from S3 or local location to watsonx.data Iceberg table:



 Ingest a single CSV/Parquet file from S3 location by using a command 

To ingest a single CSV/Parquet file from a S3 location, run the following command:

ibm-lh data-copy --source-data-files <SOURCE_DATA_FILE> --source-s3-creds "AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>,AWS_REGION=<YOUR_REGION>, BUCKET_NAME=<YOUR_BUCKET>, ENDPOINT_URL=<YOUR_ENDPOINT_URL>" --staging-location <STAGING_LOCATION> --staging-s3-creds "AWS_SECRET_ACCESS_KEY=<YOUR_TARGET_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_TARGET_ACCESS_KEY_ID>,AWS_REGION=<YOUR_TARGET_REGION>, BUCKET_NAME=<YOUR_TARGET_BUCKET>, ENDPOINT_URL=<YOUR_TARGET_ENDPOINT_URL>” --target-tables <TARGET_TABLES> --ingestion-engine-endpoint <INGESTION_ENGINE_ENDPOINT> --dbuser <DBUSER> --dbpassword <DBPASSWORD> --create-if-not-exist

For example:
Creating an ingestion job by using the configuration file 

After installing the ibm-lh tool, you can run the tool to ingest data into IBM® watsonx.data through the CLI. The configuration files and examples are listed in this topic.

To run the ingestion jobs, you can use the configuration file option. Advantage of using a configuration file is, multiple ingestions can be completed in a single starting of the ingestion tool.

Run the following command to do multiple ingestion jobs after you update the configuration file:

ibm-lh data-copy --ingest-config /<your_ingest_configfilename>

To access IBM Cloud Object Storage (COS) and MinIO object storage, specify the ENDPOINT_URL in --source-s3-creds and --staging-s3-creds to pass the corresponding url to the tool. For more information about IBM COS, see [https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints).

Replace the absolute values inside the angular brackets of the command examples or config file examples with the values applicable to your environment. See [Options and variables supported in ibm-lh tool](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-cli_commands).

Following are the details of the config file option to ingest data files from S3 or local location to watsonx.data Iceberg table:



 Ingest a single CSV/Parquet file from S3 location by using config file 

To ingest a single Parquet file from a S3 location, update the following configuration file:

[global-ingest-config]
target-tables:<TARGET_TABLES>
ingestion-engine:hostname=<hostname>,port=<port>
create-if-not-exist:<true>
[ingest-config1]
Geospatial analytics 

Geospatial SQL functions allow developers to efficiently index and perform calculations on geospatial data. The combination of geospatial analytic functions with the serverless form factor of IBM Cloud® Data Engine makes it easy to build location analytics solutions with your data in the cloud.

[Getting Started with Data Engine Geospatial](https://github.com/IBM-Cloud/data/wiki/Getting-Started-with-SQL-Query-Geospatial) shows you how to use Data Engine to analyze geospatial data.

[FordGo bike analysis using Data Engine](https://github.com/IBM-Cloud/data/wiki/FordGo-bike-analysis-using-SQL-Query) demonstrates how to use Data Engine's geospatial capabilities to analyze bike-share data from San Francisco.

[Querying geospatial data by using Data Engine](https://www.ibm.com/cloud/blog/querying-geospatial-data-using-ibm-sql-query) shows you how to create and run geospatial queries with Data Engine on bicycle location data that is stored in IBM Cloud® Object Storage by using IBM® Watson Studio and PixieDust.

See the following article that explains the key highlights of the geospatial functions of Data Engine: [Geospatial functions of IBM Cloud Data Engine](https://medium.com/@raghukganti/geospatial-without-projections-bb3334b1510c) cover the Full Earth feature of the geospatial functions in Data Engine.
: Users can [archive cold data](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archive) by setting the proper parameters in a bucket lifecycle configuration policy, either using the console, REST API, or a language-specific SDK.





 18 August 2018 

New locations!
: Buckets can now be created in a Single Data Center configuration in Sao Paolo, Brazil and Oslo, Norway. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 22 June 2018 

New locations!
: Buckets can now be created in the EU Germany region. Data stored in these buckets is distributed across three availability zones in the EU Germany region. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints) documentation.
: Buckets can now be created in a Single Data Center configuration in Chennai, India and Amsterdam, Netherlands. This allows for lower latency when accessing storage from compute resources co-located within the same data center, or for data requiring a specific geographic location. More information can be found in [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 16 March 2018 

New location!
: Buckets can now be created in an AP Cross Region configuration. Data stored in these buckets is distributed across the Seoul, Tokyo, and Hong Kong data centers. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).

New feature!
: Users can run SELECT SQL queries directly against structured data objects using IBM Cloud® Data Engine.
Following are the exception cases where staging location is required:



* Any or all of the parquet files in the folder are huge.
* Any or all of the parquet files in the folder have special columns, such as date or decimal.







 Ingest a CSV/Parquet file or folder from a local file system by using a command 

To ingest a CSV/Parquet file or a folder of files from a local file system, run the following command:

ibm-lh data-copy --source-data-files <SOURCE_DATA_FILE> --staging-location <STAGING_LOCATION> --staging-s3-creds "AWS_SECRET_ACCESS_KEY=<YOUR_TARGET_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_TARGET_ACCESS_KEY_ID>,AWS_REGION=<YOUR_TARGET_REGION>, BUCKET_NAME=<YOUR_TARGET_BUCKET>, ENDPOINT_URL=<YOUR_TARGET_ENDPOINT_URL>” --target-tables <TARGET_TABLES> --ingestion-engine-endpoint <INGESTION_ENGINE_ENDPOINT> --dbuser <DBUSER> --dbpassword <DBPASSWORD> --create-if-not-exist

For example:

ibm-lh data-copy --source-data-files /cust-bucket/warehouse/a_source_file1.parquet --staging-location s3://cust-bucket/warehouse/staging/ --staging-s3-creds "AWS_ACCESS_KEY_ID=zzzzzz,AWS_SECRET_ACCESS_KEY=vvvvvvv,AWS_REGION=us-east-1,BUCKET_NAME=cust-bucket" --target-tables iceberg_target_catalog.ice_schema.cust_tab1 --ingestion-engine-endpoint "hostname=localhost,port=8080" --create-if-not-exist
Viewer As a viewer, you can view service instances, but you can't modify them. 







 IBM Lakehouse 

Review the available platform and service roles and the actions mapped to each to help you assign access. If you're using the CLI or API to assign access, use lakehouse for the service name.

Platform roles

Service roles

Actions



Table 107. Platform roles - IBM Lakehouse
Use the tab buttons to change the context of the table. This table has row and column headers. The row headers provide the platform role name and the column headers identify the specific information available about each role.

 Role Description 

 Administrator As an administrator, you can perform all platform actions based on the resource this role is being assigned, including assigning access policies to other users. 
 Editor As an editor, you can perform all platform actions except for managing the account and assigning access policies. 
 Operator As an operator, you can perform platform actions required to configure and operate service instances, such as viewing a service's dashboard. 
 Viewer As a viewer, you can view service instances, but you can't modify them. 







 Language Translator 

Review the available platform and service roles and the actions mapped to each to help you assign access. If you're using the CLI or API to assign access, use language-translator for the service name.

Service roles

Actions



Table 108. Service roles - Language Translator
Use the tab buttons to change the context of the table. This table has row and column headers. The row headers provide the service role name and the column headers identify the specific information available about each role.

 Role Description 

 Manager As a manager, you have permissions beyond the writer role to complete privileged actions as defined by the service. In addition, you can create and edit service-specific resources. 
 Reader As a reader, you can perform read-only actions within a service such as viewing service-specific resources. 
 Writer As a writer, you have permissions beyond the reader role, including creating and editing service-specific resources. 







 IBM Log Analysis with LogDNA 

Review the available platform and service roles and the actions mapped to each to help you assign access.
Ingest multiple CSV/Parquet files and CSV folders from S3 location by using a command 

To ingest multiple CSV/Parquet files and CSV folders from a S3 location, run the following command:

ibm-lh data-copy --source-data-files <SOURCE_DATA_FILE> --source-s3-creds "AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ASSESS_KEY>, AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>,AWS_REGION=<YOUR_REGION>, BUCKET_NAME=<YOUR_BUCKET>, ENDPOINT_URL=<YOUR_ENDPOINT_URL>" --staging-location <STAGING_LOCATION> --staging-s3-creds "AWS_SECRET_ACCESS_KEY=<YOUR_TARGET_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_TARGET_ACCESS_KEY_ID>,AWS_REGION=<YOUR_TARGET_REGION>, BUCKET_NAME=<YOUR_TARGET_BUCKET>, ENDPOINT_URL=<YOUR_TARGET_ENDPOINT_URL>” --target-tables <TARGET_TABLES> --ingestion-engine-endpoint <INGESTION_ENGINE_ENDPOINT> --dbuser <DBUSER> --dbpassword <DBPASSWORD> --create-if-not-exist

For example:
Auditing events for Db2 Warehouse on Cloud 

IBM Cloud Activity Tracker allows you to forward the Db2 diagnostic log (db2diag.log) from your IBM Cloud instances to your own Log Analysis with LogDNA service instance. Choosing to enable log forwarding allows you to use Log Analysis to retain these logs for the time period required by your organization, or to search or analyze the logs.

As a security officer, auditor, or manager, you can use the IBM Cloud® Activity Tracker service to track how users and applications interact with the Db2 Warehouse on Cloud service in IBM Cloud.

IBM Cloud Activity Tracker records user-initiated activities that change the state of a service in IBM Cloud. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. In addition, you can be alerted about actions as they happen. The events that are collected comply with the Cloud Auditing Data Federation (CADF) standard. For more information, see the getting started tutorial for IBM Cloud Activity Tracker.

Activity Tracker with LogDNA integration is available for IBM® Db2® Warehouse on Cloud deployments in the regions according to the following table:



Table 1. Activity Tracker regions

 Deployment region Activity Tracker region 

 Dallas us-south 
 Washington us-east 
 Frankfurt eu-de 
 London eu-gb 
 Sydney au-syd 
 Tokyo jp-tok 
 Toronto ca-tor 





 Activity Tracker with LogDNA provisioning 

After you provision the service, events are automatically forwarded from all of your IBM® Db2® Warehouse on Cloud deployments in the same region.

The service can be provisioned from its [catalog page](https://cloud.ibm.com/catalog/services/logdnaat?callback=%2Fobserve%2Factivitytracker%2Fcreate) or from an existing [Observability Dashboard](https://cloud.ibm.com/observe/activitytracker).

The Activity Tracker with LogDNA service has a Lite plan that is free to use, but it only offers streaming events.