vania-janet/multiturn-rag-retrieval-data huggingface dataset:
_cosClient = createClient(api_key, service_instance_id, endpoint_url, location);

// Use the client to read the configuration
BucketLifecycleConfiguration config = _cosClient.getBucketLifecycleConfiguration(bucketName);

System.out.println(config.toString());
}

/
* @param bucketName
* @param clientNum
* @param api_key
* @param service_instance_id
* @param endpoint_url
* @param location
* @return AmazonS3
/
public static AmazonS3 createClient(String api_key, String service_instance_id, String endpoint_url, String location)
{
AWSCredentials credentials;
credentials = new BasicIBMOAuthCredentials(api_key, service_instance_id);

ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

}





 Delete a bucket’s lifecycle configuration, including expiration 

This implementation of the DELETE operation uses the lifecycle query parameter to examine lifecycle settings for the bucket. All lifecycle rules associated with the bucket will be deleted. Transitions defined by the rules will no longer take place for new objects. However, existing transition rules will be maintained for objects that were already written to the bucket before the rules were deleted. Expiration Rules will no longer exist.
Summary of data returned by the instance metadata service 

Use the instance metadata service to access information about the instance, SSH keys, and placement groups. This topic provides a summary of this data by URI path and metadata key, and describes the metadata.

The API response for each metadata type is presented in JSON format that contains metadata key:value pairs.



 Summary of metadata for instances 

Use the information in the Table 1 to understand the type of metadata returned for an instance. To access all metadata, see [Use the instance metadata service](https://cloud.ibm.com/docs/vpc?topic=vpc-imd-get-metadata).



Table 1. Metadata for an instance

 Instance URI path Metadata key Description of the metadata 

 /instance availability_policy The availability policy for this virtual server instance. 
 /instance bandwidth The total bandwidth in megabits per second that are shared across the virtual server instance's network interfaces and block storage volumes. 
 /instance boot_volume_attachment Boot volume attachment, with volume name, CRN, and ID. 
 /instance catalog_offering This is displayed if the virtual server instance was provisioned with a custom image in a private catalog. 
 /instance created_at The date and time that the virtual server instance was created. 
 /instance crn The Cloud Resource Name for this virtual server instance. 
 /instance dedicated_host Collection of the dedicated host references, which includes the CRN, deleted hosts, ID, name, and resource_type. 
 /instance disks A list of the instance's disks. 
 /instance gpu The virtual server instance GPU configuration 
 /instance id The unique identifier for this virtual server instance. 
 /instance image The image from which the virtual server instance was provisioned. For example, a Debian Linux stock image. 
 /instance lifecycle_reasons The reasons for the current lifecycle_state. 
 /instance lifecycle_state The lifecycle state of the virtual server instance. 
 /instance memory The amount of memory; truncated to whole gigabytes. 
 /instance metadata_service Indicates whether the metadata service endpoint is available to the virtual server instance. (Enabled status is true or false.)
The nginx server adds a header to the original request, and then calls the Store microservice. However, the initial steps do not produce any logs. To see the first logs from an incoming API request, check the logs in the Store microservice.
* Integrations: Service that supports the built-in integrations, such as web chat and preview link. Introduced with the 1.5.0 release.
* Master: Controls the lifecycle of underlying intent and entity models. This microservice is part of the language understanding pipeline. The NLU microservice uses LiteLinks to send a request to the Master microservice for a new model to be trained. This type of request occurs when a dialog skill is created or an existing dialog skill is updated. After a model is successfully trained, the TAS microservice is called to load the model.
* Numeric system entities: Manages the numeric value recognition that is used by the system entities (such as @sys-date, @sys-number, and so on). Introduced with the 1.5.0 release.
* Recommends: Supports recommendations from Watson, such as dictionary-based entity synonyms and intent conflicts. Used only at authoring time from the web UI. When a user requests synonym recommendations for an entity, for example, the UI microservice calls the Store microservice. The Store microservice forwards the request to the Recommends microservice. The Recommends microservice looks up the synonyms for the current word by using the embeddings that are stored in MongoDB, and stores them in a Redis cache. This microservice sends a response with a list of synonyms to the Store microservice. A similar workflow is used to identify intent conflicts in a dialog skill.
* Skill-search: Manages API calls to a Discovery service instance that is enabled in the same cluster. When a v2 /message API call reaches the Store microservice, the Store retrieves session state information from Redis and start processing skills. When the assistant that is being processed has a search skill, the Store microservice calls this microservice over HTTPS REST.
* @param service_instance_id
* @param endpoint_url
* @param location
* @return AmazonS3
/
public static AmazonS3 createClient(String api_key, String service_instance_id, String endpoint_url, String location)
{
AWSCredentials credentials;
credentials = new BasicIBMOAuthCredentials(api_key, service_instance_id);

ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

public static List contentsForDeletion(String bucketName, int maxKeys) {
System.out.printf("Retrieving bucket contents (V2) from: %sn", bucketName);

boolean moreResults = true;
String nextToken = "";

while (moreResults) {
ListObjectsV2Request request = new ListObjectsV2Request()
.withBucketName(bucketName)
.withMaxKeys(maxKeys)
.withContinuationToken(nextToken);

ListObjectsV2Result result = _cosClient.listObjectsV2(request);
for(S3ObjectSummary objectSummary : result.getObjectSummaries()) {
for MICROSERVICE in analytics clu-embedding clu-serving clu-training create-slot-job data-governor dialog dragonfly-clu-mm ed es-store etcd integrations master nlu recommends sireg-ubi-ja-tok-20160902 sireg-ubi-ko-tok-20181109 spellchecker-mm store store-admin store-cronjob store-sync store_db_creator store_db_schema_updater system-entities tfmm ui ${INSTANCE}-redis webhooks-connector
do
 Get original networking policy
 Clean up metadata fields to get the resource applied by Watson Assistant
 Change name and add component to selector
 Apply to the cluster
oc get networkpolicy $INSTANCE-network-policy --output yaml | oc patch --dry-run=client --output=yaml -f - --type=json --patch='[
{"op":"remove", "path":"/metadata/creationTimestamp"},
{"op":"remove", "path":"/metadata/generation"},
{"op":"remove", "path":"/metadata/resourceVersion"},
{"op":"remove", "path":"/metadata/uid"},
{"op":"remove", "path":"/metadata/annotations/kubectl.kubernetes.io1last-applied-configuration"},
{"op":"remove", "path":"/metadata/ownerReferences"}
]' | oc patch --dry-run=client --output=yaml -f - --type=merge --patch "{
"metadata": {"name": "${INSTANCE}-network-policy-$( echo $MICROSERVICE | tr _ -)"},
"spec": {"podSelector":{"matchLabels":{"component": "${MICROSERVICE}"}}}
}"
Some of the metadata is collected directly from the customer and other metadata is generated as an artifact of the automation logic that is used to order and configure the instance such as the hosts, storage, networking, and service data. Instance metadata includes:



* Client cloud account and contact information
* Instance names and IDs of VMware® instances that are deployed for customers
* VMware instances locations
* Installed services, for example, Veeam®, Zerto, or HCX™
* Deployment state of each instance, for example Deploying, Failed, or Ready
* Configuration of user solutions and the underlying components that are used in customer solutions (Compute, networking, storage, licenses)
* Credentials to access the underlay IaaS and VMware components that are hosting the customers' workloads
* Management logs generated from automation
* Support data that is associated with helping the client resolve issues and questions



VMware as a Service does not collect any client or special personal information.





 About customer-managed keys 

Customer keys (BYOK) are not initially supported. VMware as a Service generates the keys uniquely per instance.







 Deleting your data in VMware as a Service 



 Deleting VMware as a Service metadata 

Client metadata is associated with the VMware instances that the client creates. When the VMware instance is deleted by the client that the metadata is maintained in the DB where the instance state is set to deleted. Client metadata is maintained in the DB unless the client specifically requests that the data is removed by using a support ticket.

Logs and support data that is associated with the instance are retained by IBM Cloud policy for 1 year unless requested for deletion through an IBM Support ticket.





 Deleting VMware as a Service customer data 

IBM Cloud for VMware as a Service customer data is deleted including any customer backups of data by using the service in association with the deletion of the service instance. The cloud service provided data encryption key used by customers to encrypt data is destroyed on deletion. This event renders the data unusable there forward.







 Related links
For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).







 Configuring Analytics Engine instance by using IBM Cloud® console 

To configure your Analytics Engine instance from the IBM Cloud® Resource list, complete the following steps:



1. Log in to your IBM Cloud® account.
2. Access the [IBM Cloud® Resource list](https://test.cloud.ibm.com/resources).
3. Search your Analytics Engine instance and click the instance to see the details.
4. Click Manage > Configuration to view the configuration.
5. In the Default Spark configuration section, click Edit.
6. Add the following configuration to the Default Spark configuration section.

spark.sql.catalogImplementation = hive
spark.driver.extraClassPath = /opt/ibm/connectors/iceberg-lakehouse/iceberg-3.3.2-1.2.1-hms-4.0.0-shaded.jar
spark.sql.extensions = org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.iceberg.vectorization.enabled = false
spark.sql.catalog.lakehouse = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lakehouse.type = hive
spark.sql.catalog.lakehouse.uri = <hms-thrift-endpoint-from-watsonx.data> for example (thrift://81823aaf-8a88-4bee-a0a1-6e76a42dc833.cfjag3sf0s5o87astjo0.databases.appdomain.cloud:32683)
spark.hive.metastore.client.auth.mode = PLAIN
spark.hive.metastore.client.plain.username = <hms-user-from-watsonx.data> (for example, ibmlhapikey)
Retrieving Hive metastore(HMS) credentials 

Complete the following steps to generate HMS credentials. HMS credential includes:



*  API key. Follow the [steps](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hmshms_api) to generate the key.
*  HMS endpoint (thrift url). Follow the [steps](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hmshms_url) to generate the thrift url.
*  HMS username is by default ibmlhapikey





  Generating API key 



1.  In the IBM Cloud console, go to Manage and create service ID. For information about creating service ID, see [Creating a service ID](https://ondeck.console.cloud.ibm.com/docs/account?topic=account-serviceids&interface=uicreate_serviceid).
2.  In the Access tab, from Access policies, click Assign access. The Assign access to Doc Service page opens.
3.  Select Access policy. In the Create policy section, select the UI tab.
4.  Search and select watsonx.data from the Service field.
5.  Click Next. The Access policies section is reflected with the role.
6.  Go to Manage > Access (IAM), and select the Service IDs.
7.  Identify the row of the service ID for which you want to select an API key. Select the name of the service ID.
8.  Click API keys.
9.  Click Create.
10. Specify a name and description to easily identify the API key.
11. Click Create.
12. Save your API key by copying or downloading it to secure location.







  Getting the HMS endpoint 



1.  Log in to watsonx.data console.
2.  From the navigation menu, select Infrastructure manager.
3.  Select the catalog.
4.  Click Details to view the catalog details page.
5.  Copy the Metastore host. This is your HMS endpoint (thrift url).
public static AmazonS3 createClient(String api_key, String service_instance_id, String endpoint_url, String location)
{
AWSCredentials credentials;
credentials = new BasicIBMOAuthCredentials(api_key, service_instance_id);

ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cosClient = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

// Create a new bucket
public static void createBucket(String bucketName, AmazonS3 cosClient)
{
cosClient.createBucket(bucketName);
System.out.printf("Bucket: %s created!n", bucketName);
}

// Retrieve the list of available buckets
public static void listBuckets(AmazonS3 cosClient)
{
System.out.println("Listing buckets:");
final List<Bucket> bucketList = _cosClient.listBuckets();
for (final Bucket bucket : bucketList) {
System.out.println(bucket.getName());
}
System.out.println();
}

// Retrieve the list of contents for a bucket
public static void listObjects(String bucketName, AmazonS3 cosClient)
{
System.out.println("Listing objects in bucket " + bucketName);
2022/02/24 22:25:59 Terraform apply | 335: resource "ibm_is_instance" "custom_image_instance" {
2022/02/24 22:25:59 Terraform apply 
2022/02/24 22:25:59 [ 1m31mTerraform APPLY error: Terraform APPLY errorexit status 139m0m
When the solution was provisioned, a profile with instance storage was selected. However, instance storage is not supported for VPC. Instance storage profiles have a 'd' in the profile name, for example 'cx2d-2x4' is a compute profile that has instance storage, 'cx2-2x4' is a compute profile that does not have instance storage.Either edit the profile and remove the 'd' or enter a new profile. For more information on available instance profiles, see Instance profiles]] ! ! .<-- </section "id="section-troubleshooting-instance-storage" "> --><-- <section "id="section-troubleshooting-vpc-cc-not-registered" "> --> Cloud connector is not registering During post-provisioning, you access the Cloud Connector instance]] ! ! ! ! ! ! . If the Cloud Connector is not registering, in can be one of two problems. The messages are logged in the Cloud Connector log file on Citrix.Create a floating ip to use Remote Desktop to get to the cloud connector log file on Citrix cloud. The file is C:ProgramDataIBMCitrixDaaSLogsIBMCitrixDaaSInstallation.log. Close to the end of the log file is either a:<-- <ul> --> * 1603 error * ResourceLocationId error<-- </ul> -->If the error is 1603, contact Citrix support.If the error is ResourceLocationId, an invalid resource location was used during provisioning, for example, the Resource location might have been removed or a limit was reached. To resolve this error, try one of the following actions:<-- <ol> -->1.
GET https://{bucket-name}.{endpoint}?location virtual host style

Example request

This is an example of retrieving the location of the "apiary" bucket.

GET /apiary?location= HTTP/1.1
Authorization: Bearer {token}
Host: s3.us.cloud-object-storage.appdomain.cloud

Example request

GET /apiary?location= HTTP/1.1
Authorization: 'AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;,Signature={signature}'
x-amz-date: {timestamp}
Content-Type: text/plain
Host: s3.us.cloud-object-storage.appdomain.cloud

Example response

HTTP/1.1 200 OK
Date: Tue, 12 Jun 2018 21:10:57 GMT
X-Clv-Request-Id: 0e469546-3e43-4c6b-b814-5ad0db5b638f
Accept-Ranges: bytes
Server: Cleversafe/3.13.3.57
X-Clv-S3-Version: 2.5
x-amz-request-id: 0e469546-3e43-4c6b-b814-5ad0db5b638f
Content-Type: application/xml
Content-Length: 161

<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
us-south-standard
</LocationConstraint>

--------------------





 Create a bucket lifecycle configuration 

A PUT operation uses the lifecycle query parameter to set lifecycle settings for the bucket. A Content-MD5 header is required as an integrity check for the payload.

Not all operations are supported in Satellite environments.
ClientConfiguration clientConfig = new ClientConfiguration().withRequestTimeout(5000);
clientConfig.setUseTcpKeepAlive(true);

AmazonS3 cos = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();

return cos;
}
Show more

Key Values



* <endpoint> - public endpoint for your cloud Object Storage (available from the [IBM Cloud Dashboard](https://cloud.ibm.com/resources)). For more information about endpoints, see [Endpoints and storage locations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpointsendpoints).
* <api-key> - api key generated when creating the service credentials (write access is required for creation and deletion examples)
* <resource-instance-id> - resource ID for your cloud Object Storage (available through [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-idt-cli) or [IBM Cloud Dashboard](https://cloud.ibm.com/resources))
* <location> - default location for your cloud Object Storage (must match the region that is used for <endpoint>)



SDK References



* Classes



* [AmazonS3ClientBuilder](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3ClientBuilder.html)
SERVICE_INSTANCE = os.environ.get('SERVICE_INSTANCE_ID')
ENDPOINT = os.environ.get('ENDPOINT')

BUCKET = "my-replication-bucket" The bucket that will enable replication.

 Create resource client with configuration info pulled from environment variables.
cosClient = ibm_boto3.client("s3",
ibm_api_key_id=API_KEY,
ibm_service_instance_id=SERVICE_INSTANCE,
config=Config(signature_version="oauth"),
endpoint_url=ENDPOINT
)

response = cosClient.put_bucket_versioning(
Bucket=BUCKET,
ReplicationConfiguration={
'Rules': [
{
'ID': 'string',
'Priority': 123,
'Filter': {
'Prefix': 'string',
'Tag': {
'Key': 'string',
'Value': 'string'
},
'And': {
'Prefix': 'string',
'Tags':
{
'Key': 'string',
'Value': 'string'
},
]
}
},
'Status': 'Enabled'|'Disabled',
'Destination': {
'Bucket': 'string',
},
'DeleteMarkerReplication': {
'Status': 'Enabled'|'Disabled'
}
},
]
}
)
Show more

Listing the versions of an object using the same client:

resp = cosClient.list_object_versions(Prefix='some-prefix', Bucket=BUCKET)

Note that the Python APIs are very flexible, and there are many different ways to accomplish the same task.





 Node.js
When you create a new table with a Presto Hive connector that uses an S3 folder from an external location, Presto does not recognize the path as a directory and an error might occur.

For example, when creating a customer table in the target directory DBCERT/tbint in a bucket that is called dqmdbcertpq by using the IBM Cloud UX and Aspera S3 console, the following error is encountered: External location must be a directory.

CREATE TABLE "hive-beta"."dbcert"."tbint" (
RNUM int , CBINT bigint
) WITH (
format='PARQUET', external_location = 's3a://dqmdbcertpq/DBCERT/tbint'
);
Query 20230509_113537_00355_cn58z failed: External location must be a directory

Objects in a file system are stored as objects and their path. The object and path must have an associated metadata. If the path is not associated with the metadata, Presto fails to recognize the object and responds that the path is not a directory.





 Issue: Assigning Grant or Revoke privilege 

Assigning Grant or Revoke privilege to a user through access policy does not work as expected in the following scenarios:



1. User_A adds a bucket and a Hive catalog (for example, useracat02).
2. User_A creates a schema and a table.
3. User_B and User_C are assigned User role to the catalog.
4. User_A adds allow grant policy to User_B
5. User_B connects to the catalog and runs grant select to User_C.

presto:default> grant select on useracat02.schema_test_01.tab_1 to "6ff74bf7-b71b-42f2-88d9-a98fdbaed304";
6. When the User_C connects to the catalog and runs select command on the table, the command fails with access denied message.
<resource-instance-id> Unique ID for the Service Instance crn:v1:bluemix:public:cloud-object-storage:global:a/xx999cd94a0dda86fd8eff3191349999:9999b05b-x999-4917-xxxx-9d5b326a1111:: 
 <storage-class> Storage class for a new bucket us-south-standard 



For more information about endpoints, see [Endpoints and storage locations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpointsendpoints).

Code examples are tested on supported release versions of Python.



* Java
* Python
* Go
* Node



// Required libraries
import com.ibm.cloud.objectstorage.ClientConfiguration;
import com.ibm.cloud.objectstorage.SDKGlobalConfiguration;
import com.ibm.cloud.objectstorage.SdkClientException;
import com.ibm.cloud.objectstorage.auth.AWSCredentials;
import com.ibm.cloud.objectstorage.auth.AWSStaticCredentialsProvider;
import com.ibm.cloud.objectstorage.client.builder.AwsClientBuilder;
import com.ibm.cloud.objectstorage.oauth.BasicIBMOAuthCredentials;
import com.ibm.cloud.objectstorage.services.s3.AmazonS3;
import com.ibm.cloud.objectstorage.services.s3.AmazonS3ClientBuilder;
import com.ibm.cloud.objectstorage.services.s3.model.;
import com.ibm.cloud.objectstorage.services.s3.transfer.TransferManager;
import com.ibm.cloud.objectstorage.services.s3.transfer.TransferManagerBuilder;
import com.ibm.cloud.objectstorage.services.s3.transfer.Upload;