vania-janet/multiturn-rag-retrieval-data huggingface dataset:
_cosClient = createClient(api_key, service_instance_id, endpoint_url, location);

// Delete the configuration.
_cosClient.deleteBucketLifecycleConfiguration(bucketName);

// Verify that the configuration has been deleted by attempting to retrieve it.
config = _cosClient.getBucketLifecycleConfiguration(bucketName);
String s = (config == null) ? "Configuration has been deleted." : "Configuration still exists.";
System.out.println(s);
}

/
* @param bucketName
* @param clientNum
* @param api_key
* @param service_instance_id
* @param endpoint_url
* @param location
* @return AmazonS3
/
public static AmazonS3 createClient(String api_key, String service_instance_id, String endpoint_url, String location)
{
AWSCredentials credentials;
credentials = new BasicIBMOAuthCredentials(api_key, service_instance_id);

ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

}

Show more







 Next Steps 

Expiration is just one of many lifecycle concepts available for IBM Cloud Object Storage.
_cosClient = createClient(api_key, service_instance_id, endpoint_url, location);

// Delete the configuration.
_cosClient.deleteBucketLifecycleConfiguration(bucketName);

// Verify that the configuration has been deleted by attempting to retrieve it.
config = _cosClient.getBucketLifecycleConfiguration(bucketName);
String s = (config == null) ? "Configuration has been deleted." : "Configuration still exists.";
System.out.println(s);
}

/
* @param bucketName
* @param clientNum
* @param api_key
* @param service_instance_id
* @param endpoint_url
* @param location
* @return AmazonS3
/
public static AmazonS3 createClient(String api_key, String service_instance_id, String endpoint_url, String location)
{
AWSCredentials credentials;
credentials = new BasicIBMOAuthCredentials(api_key, service_instance_id);

ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

}







 Next Steps 

Expiration is just one of many lifecycle concepts available for IBM Cloud Object Storage.
This applies to the IBM Cloud Object Storage home instance as well as endpoints used from your applications (either your code or what you pass as parameters in the configurations at instance level or application level). Direct endpoints provide better performance than public endpoints and do not incur charges for any outgoing or incoming bandwidth.<br><br><br> <br><br> * Disaster Recovery (DR) Resiliency: [IBM Cloud Object Storage documentation.](https://cloud.ibm.com/docs/cloud-object-storage/info?topic=cloud-object-storage-endpointsendpoints)<br> * Encryption: [Getting started with encryption keys](https://cloud.ibm.com/docs/key-protect?topic=key-protect-getting-started-tutorialgetting-started-tutorial) and [Object Storage manage encryption](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-encryptionencryption)<br> * Service credentials: [Service credentials](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-service-credentialsservice-credentials)<br> * Direct endpoints for IBM Cloud Object Storage: [Endpoints and storage locations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints)<br><br><br> 
 Use private endpoints for the external Hive metastore If you are using Spark SQL and want to use an external metastore such as use IBM Cloud Databases for PostgreSQL as your Hive metastore, you must use the private endpoint for the database connection for better performance and cost savings. <br><br> * [Working with Spark SQL and an external metastore](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-external-metastore)<br><br><br> 
 Running applications with resource overcommitment There is a quota associated with each Analytics Engine Serverless instance. When applications are submitted on an instance, they are allocated resources from the instance quota.
DESCRIBE METAINDEX ON cos://us-geo/sql/metergen STORED AS parquet







 Show indexes 



 showIndexes 

List all stored indexes in the base location. Tables with a different index location are not displayed in the list.

-- list all Metaindexes in the base location
SHOW METAINDEXES







 Alter index 



 alterIndex 

You must alter the Object Storage location for all indexes only once to define the base location. If you change it later, Data Engine cannot find the index metadata anymore. Existing index metadata on previous location is not dropped. Therefore, you can always switch back to the old location when needed.

-- set the default location for all indexes
ALTER METAINDEX SET LOCATION cos://us-south/<mybucket>/<mypath>/







 Alter table set location 



 alterTableSetLocation 

With this command, you can define a location for this specified Hive table. If you change it later, Data Engine does not find the index metadata anymore. Existing index metadata on previous location is not dropped, therefore you can always switch back to the old location when needed.

-- set the index location for the table CUSTOMERS_PARTITIONED
ALTER TABLE CUSTOMERS_PARTITIONED SET METAINDEX LOCATION cos://us-south/<mybucket>/<mypath>







 Alter table drop location 



 alterTableDropLocation 

With this command, you can drop a location for the specified table. Use this command if the index metadata is to be fetched from the base location. The metadata for the index that is stored in Object Storage is not dropped and must be cleaned up manually.

-- set the index location for the table CUSTOMERS_PARTITIONED
ALTER TABLE CUSTOMERS_PARTITIONED DROP METAINDEX LOCATION







 IndexAsset 



 metaindexAsset 

The indexAsset is either based on a table or Cloud Object Storage location.

The metaindexAssetLocation is a subset of the [externalTableSpec](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec).
* The Container level compute workload isolation section represents the IBM-managed Delivery Pipeline shared workers component of the service. This component supports single tenant workloads.
* The multi-tenant and single tenant workloads each run in different sets of clusters, and are compute-isolated from each other.
* Each Delivery Pipeline job or step runs in its own container that is compute-isolated from the other containers within a cluster. This isolation applies to both IBM-managed shared workers and customer-managed shared workers.



Zoom

![Continuous Delivery compute workloads](https://cloud.ibm.com/docs-content/v1/content/562806c0b18119f4d21cb66f5ae6d051634b4a16/ContinuousDelivery/images/CD_Compute_Arch.png)

Figure 2. Continuous Delivery compute workloads





 Continuous Delivery dependencies 

The Continuous Delivery service depends on several IBM Cloud Platform components and services that include, but are not limited to, the following tasks.



* Hosting the internal microservices of the Continuous Delivery service
* Integrating into the IBM Cloud Platform and user interface
* Storing service and customer-owned data
* Logging and auditing service events
* Backing up service and customer data regularly



The following table lists the main dependencies of the Continuous Delivery service.

The Continuous Delivery service launches its dependencies over public endpoints.



Table 2. Continuous Delivery dependencies

 Dependency Type Purpose 

 Console Platform Enables users to manage Continuous Delivery service instances and toolchains by way of a web UI. Handles user login for most Continuous Delivery service web pages. 
 Identity and Access Management (IAM) Platform Provides authorization checks to Continuous Delivery and toolchain resources and actions. Generates and authenticates IAM API keys such as those that are used by the Delivery Pipeline component. 
 Search and tagging Platform Provides cached, high performance lookup of resources. 
 Resource controller Platform Coordinates the provisioning, updating, and de-provisioning of Continuous Delivery and toolchain resources. 
 Usage metering Platform Receives usage metrics from the Continuous Delivery service to report usage and compute bills for customer accounts.
<br>Client metadata such as Client email ID, workspace, action names. Customer-owned content such as Agent data, Agent policies. <br>Client metadata such as Agent location, Agent name. 
 Applications Customer-owned software components, such as executables, web applications, middleware, frameworks, libraries, and other software packages that the client developed or acquired by third parties and deployed in IBM Cloud. None Customer-owned software components installed in Agents runtime. 
 Service instance An entity that consists of resources that are reserved for a particular service. Schematics service instance such as Workspaces, Actions, Inventories, Blueprints Schematics service instance such as Agents instance. 
 Operating systems The Operating System software and configuration that are deployed in virtual or bare metal servers, such as Linux, Windows, or similar to the ones provided in stock images. [Universal Base Image (UBI-8)](https://catalog.redhat.com/software/containers/ubi8/ubi/5c359854d70cc534b3a3784e) [Universal Base Image (UBI-8)](https://catalog.redhat.com/software/containers/ubi8/ubi/5c359854d70cc534b3a3784e) 
 Virtual and bare metal servers The virtual or bare metal servers that are ordered and managed through IBM Cloud services. IBM owns the IKS used by Schematics. Client manages the [IKS / ROKS / Kubernetes](https://cloud.ibm.com/docs/containers?topic=containers-responsibilities_iks) cluster where Agents are deployed. 
 Virtual storage The block, file, or Object Storage buckets ordered and managed through IBM Cloud. IBM owns Cloudant, COS, RabbitMQ, Redis - used by Schematics Client owns and manages the instances. IBM owns the IKS local storage used by Schematics Agents COS that are used by Schematics Agents 
 Virtual network Network resources such as VLAN, VPC, subnets, or IPs provided by classic infrastructure and VPC services that are ordered and managed through IBM Cloud.
* Hive-style partitions have an eye-catching naming scheme because the column names that are used for partitioning are part of the partition object prefix, for example, /order/COUNTRY=USA/part-m-00000.snappy.parquet.
* Hive-style partitions do not contain any values for partition columns since their values are stored in the object prefix of the partition. Thus, if you copy a HIVE-style partition and rename the object prefix by removing the partition column values, you lose data.
* Hive-style partitions have a tendency for data skewing. For example, the partition that represents order data from Malta is likely much smaller than the partition that represents order data from the US. You can partition the query result into separate objects if you want to have equally sized partitions.







 Partition by columns into objects 

By partitioning a query result into objects, you can specify the exact number of equally sized result partitions. With this partitioning, you can experimentally fine-tune the number of objects to meet certain criteria for partition size. To specify the number of partitions, use the PARTITIONED INTO x BUCKETS/OBJECTS clause.

For example, knowing the size of the query result, it is possible to calculate the number of objects to end up with partitions that have a certain size. For example, 128 MB, which is the Hadoop default file size, or any other size that meets application requirements.

The INTO x BUCKETS/OBJECTS clause can be combined with the BY (column-list) clause to create some partitions that support data affinity regarding specified partition columns.

Continue with the preceding example that specifies PARTITION BY (customerid) INTO 10 OBJECTS stores the query result into 10 objects, which ensures that all data for a customer is stored in the same partition. Although it is ensured that all data for a certain customer is stored in the same partition, it is not ensured that the data is physically sorted according to the specified column.





 Partition by number of rows 

By partitioning by number of rows you can specify the number of rows that go into a single partition. To specify the number of rows stored in each partition, use the EVERY x ROWS clause.
For details, see [supported Satellite APIs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-apis-cos-satellite)



 Using PUT to update metadata 

The PUT request requires a copy of existing object as the contents are overwritten.

Syntax

PUT https://{endpoint}/{bucket-name}/{object-name} path style
PUT https://{bucket-name}.{endpoint}/{object-name} virtual host style

Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain; charset=utf-8
Host: s3.us.cloud-object-storage.appdomain.cloud
x-amz-meta-key1: value1
x-amz-meta-key2: value2

Content-Length: 533

The 'queen' bee is developed from larvae selected by worker bees and fed a
substance referred to as 'royal jelly' to accelerate sexual maturity. After a
short while the 'queen' is the mother of nearly every bee in the hive, and
the colony will fight fiercely to protect her.

Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain; charset=utf-8
Content-MD5: M625BaNwd/OytcM7O5gIaQ==
Host: s3.us.cloud-object-storage.appdomain.cloud
x-amz-meta-key1: value1
x-amz-meta-key2: value2

Content-Length: 533

The 'queen' bee is developed from larvae selected by worker bees and fed a
substance referred to as 'royal jelly' to accelerate sexual maturity. After a
short while the 'queen' is the mother of nearly every bee in the hive, and
the colony will fight fiercely to protect her.

Example response

HTTP/1.1 200 OK
: The VPC Instance Metadata service is now generally available in all regions. This free service provides a REST API that you invoke within an instance to get information about that instance. Access to the API is unavailable from outside the instance. The service is disabled by default. Before you can access the metadata, you need to use the service to generate an instance identity access token for accessing the metadata service. You can optionally get an IAM token from this token to access all IAM-enabled services. For more information, see [About VPC Instance Metadata](https://cloud.ibm.com/docs/vpc?topic=vpc-imd-about).
: Virtual server instances created on LinuxONE (s390x processor architecture) are not enabled for the VPC Instance Metadata service. The metadata service is supported only on x86 systems.

UI Update
: A new Location cascading selector is available. For example, when you provision a virtual server, the Location section now has three new cascading selector boxes. You can now choose the geographic location, the metro (region) location, and the zone. The options available in these boxes are updated that are based on your selections. If you select “Americas” for the Geography, then the Metro box displays only the Metros available for the Americas.





 22 February 2022 

Host failure recovery policies
: Host failure recovery policies for VPC are generally available (GA). For more information about host failure recovery policies, see [Host failure recovery policies](https://cloud.ibm.com/docs/vpc?topic=vpc-host-failure-recovery-policies&interface=ui).





 15 February 2022 

LinuxONE (s390x processor architecture)
: You can now create virtual server instances on LinuxONE (s390x processor architecture) in IBM Cloud in the Japan (Tokyo), United Kingdom (London), and São Paulo regions. For more information about available LinuxONE (s390x processor architecture) profiles, see [s390x instance profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-vs-profiles).
* Hms-thrift-endpoint-from-watsonx.Data: Specify the credentials for watsonx.data. For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).
* Hms-user-from-watsonx.Data: The watsonx.data username. For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).
* Hms-password-from-watsonx.Data: The watsonx.data password. For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).



To view logs of Spark application ran on IBM Analytics Engine you have to enable logging. For more information, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).
AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

}







 Next Steps 

Expiration is just one of many lifecycle concepts available for IBM Cloud Object Storage. Each of the concepts we've covered in this overview can be explored further at the [IBM Cloud Platform](https://cloud.ibm.com/).
Object Storage plug-in license license Config Object storage plug-in license: Apache license Version 2.0. Set to true to accept the license and install the plug-in. true N/A 
 Object Storage provider s3provider Config Available providers are IBM, AWS and Wasabi. For providers other than these, you must provide the Object Storage service endpoint parameter. false N/A 
 Object Storage region cos-storageclass Config Enter the region where your object storage is located. For IBM COS regions, see [https://ibm.biz/cos-endpoints-list](https://ibm.biz/cos-endpoints-list). For Wasabi, see [https://ibm.biz/wasabi-endpoints](https://ibm.biz/wasabi-endpoints). For AWS, see [https://ibm.biz/aws-endpoints](https://ibm.biz/aws-endpoints). true N/A 
 Object Storage service endpoint cos-endpoint Config Object Storage service endpoint. Required when using Object Storage providers other than IBM, AWS or Wasabi. Preference is given to Object Storage provider when both are set. false N/A 









 Storage class reference for IBM Cloud Object Storage 



Table 2. Cloud Object storage class reference

 Storage class name Volume binding mode Retain 

 ibm-s3fs-cos Immediate False 
 ibm-s3fs-cos-perf Immediate False 







 Getting help and support for IBM Cloud Object Storage 

When you use IBM Cloud Object Storage, try the following resources before you open a support case.



1. Review the FAQs in the [Block Storage for VPC docs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-faq).
2. Review the [troubleshooting documentation](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-must-gather) to troubleshoot and resolve common issues.
3.
ibm-lh data-copy --source-data-files <SOURCE_DATA_FILE> --source-s3-creds "AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>,AWS_REGION=<YOUR_REGION>, BUCKET_NAME=<YOUR_BUCKET, ENDPOINT_URL=<YOUR_ENDPOINT_URL>>" --target-tables <TARGET_TABLES> --ingestion-engine-endpoint <INGESTION_ENGINE_ENDPOINT> --dbuser <DBUSER> --dbpassword <DBPASSWORD> --create-if-not-exist

For example:

ibm-lh data-copy --source-data-files s3://cust-bucket/warehouse/ --source-s3-creds "AWS_ACCESS_KEY_ID=xxxxxx,AWS_SECRET_ACCESS_KEY=yyyyyy,AWS_REGION=us-east-1,BUCKET_NAME=cust-bucket" --target-tables iceberg_target_catalog.ice_schema.cust_tab1 --ingestion-engine-endpoint "hostname=localhost,port=8080" --create-if-not-exist

In general, this option does not require a staging location. However, there are a few exception scenarios when a staging location must be specified. When the staging location is not used, ensure that the hive catalog configured with Presto can be used with source-data-files location. Following are the exception cases where staging location is required:



* Any or all of the parquet files in the folder are huge.
* Any or all of the parquet files in the folder have special columns, such as date or decimal.







 Ingest a CSV/Parquet file or folder from a local file system by using a command 

To ingest a CSV/Parquet file or a folder of files from a local file system, run the following command:
instance is.instance.disk.update Instance storage disk name was updated 
 instance is.instance.disk.wipe Instance storage disk was wiped clean 
 instance is.instance.gpu.wipe Memory was wiped on the GPU for the Instance 







 Instance Metadata service events 

The instance metadata service events are undergoing changes and should not be used for automation. They are useful, however, for audit purposes.



Table 14. Actions that generate events for the Instance Metadata service

 Resource Action Description 

 Instance Metadata is.metadata.jwt-token.create An instance identity token was created for accessing the Metadata service 
 Instance Metadata is.metadata.computeresource-token.request Synchronizes the metadata service and IAM activity tracker events when the requested instance identity token is created. 
 Instance Metadata is.metadata.instance.initialization Initialization information was retrieved for the calling instance 
 Instance Metadata is.metadata.instance.read Metadata information was retrieved for the calling instance 
 Instance Metadata is.metadata.instance-key.get A public SSH key was retrieved for the calling instance 
 Instance Metadata is.metadata.instance-key.list Public SSH keys were listed for the calling instance 
 Instance Metadata is.metadata.instance-network-interface.get Network interface information was retrieved for the calling instance 
 Instance Metadata is.metadata.instance-network-interface.list Network interfaces were listed for the calling instance 
 Instance Metadata is.metadata.instance-placement-group.get Placement group information was retrieved for the calling instance 
 Instance Metadata is.metadata.instance-placement-group.list Placement group information was listed for the calling instance 
 Instance Metadata is.metadata.instance-volume-attachment.get A volume attachment was retrieved for the calling instance 
 Instance Metadata is.metadata.instance-volume-attachment.list Volume attachments were listed for the calling instance 







 Bare metal server events 

Some fields for Bare Metal Servers for VPC AT events will be changing between the Beta and Limited Available (LA) releases. For more information, see [Analyzing events](https://cloud.ibm.com/docs/vpc?topic=vpc-at-eventsat_events_iam_analyze).



Table 15. Actions that generate events for Bare Metal Server

 Resource Action Description