vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Create your first IBM Cloud Object Storage bucket.



1. In the navigation on the service details page, click Buckets.
2. Click Create bucket. A dialog box is displayed.
3. Enter a unique name for your bucket. The name must be unique within IBM Cloud Object Storage across all regions and across all IBM Cloud accounts.
4. From the Resiliency list, select the level of availability that you want for your data. For more information, see [IBM Cloud Object Storage regions and endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpointsendpoints).
5. Change the Location to the region where you want to store your data. Keep in mind that your data might not be allowed to be stored in every region due to legal reasons.
6. Click Create.



4. Retrieve the IBM Cloud Object Storage host name for your bucket.



1. Click on your bucket name that you created in the previous step.
2. In the navigation on the service details page, click Buckets > Configuration.
3. Note the public URL that you can use to access the data in your bucket.





For more information about configuring your service instance, see the [IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage) documentation.





 Using IBM Cloud Object Storage to back up and restore PVC data 

You can use the IBM Cloud Backup Restore Helm chart to back up data in a file storage or block storage PVC to IBM Cloud Object Storage, or restore data from IBM Cloud Object Storage to a PVC in your cluster.

Before you begin:



* Make sure that you have a PVC that you can back up or restore data to.
Login to your IBM Cloud account.
2. Navigate to Resource List in the menu.
3. Navigate to Storage in the Resource list.
4. Click the IBM Cloud Object Storage name that will display your IBM Cloud Object Storage console.
5. In the IBM Cloud Object Storage console, navigate to Buckets in the menu.
6. Click the required Bucket name to view the Bucket configuration details.
7. Navigate to Endpoints section and copy your public or private endpoints as required. Use this value as the Endpoint in the [Configuring a IBM Cloud Object Storage destination](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-destinations-cloud-object-storageen-destinations-cos-configure) section.
Using IBM Cloud Object Storage buckets as an evidence locker 

You can configure IBM Cloud Object Storage buckets to store the evidence that is generated by the compliance checks that are integrated into the DevSecOps pipelines.

The compliance automation pipeline stores the following information in your [evidence locker](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-evidencedevsecops-v2-evidence-lockers):



* Task artifacts: Test results, scan results, or any saved output by tasks.
* Task logs: After the pipeline runs, the logs for that run are sent to the evidence locker.
* Evidence: Information about tasks and their result output, which can be either a failure or success.



For more information about the format of the evidence that is sent, see [Evidence summary](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-evidence).



 Configuring your bucket 

To configure your Cloud Object Storage bucket to act as a compliance evidence locker as part of a continuous integration or continuous deployment pipeline, you can use the following information as a guide. The pipeline or toolchain template scripts do not set up the locker in Cloud Object Storage.

It is recommended that you set up a dedicated Cloud Object Storage instance for compliance-related storage as evidence lockers must be created in-boundary to your applications. [Learn more](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-cos-bucket-evidencecd-devsecops-cos-bucket-resiliency).





 Naming your bucket 

Cloud Object Storage object names consist of the following components:

{NAMESPACE} / {PIPELINE_RUN_ID} / {TYPE} / {FILE_NAME} _ {HASH}



 Examples:
Loading data from IBM Cloud Object Storage 



 Using Console 

You can load data from IBM Cloud Object Storage (COS) into IBM® Db2® on Cloud by using the Console.



 Create the necessary credentials on the COS bucket to allow Console to access the data 



1. Access the COS Bucket on the IBMCloud Dashboard by clicking on the name



Zoom

![Resource List](https://cloud.ibm.com/docs-content/v1/content/2a59ee244ed61eb35ab50e45757b42d22f720132/Db2onCloud/images/load.png)

Figure 1. Resource List





 Create Credentials on the COS bucket so Db2 Console can access the data. 



1. Select Service Credentials
2. Click on New Credentials
3. Enter Name for the Service Credential
4. Select the appropriate role
5. Ensure Include HMAC Credential is enabled



Zoom

![Create Service Credentials](https://cloud.ibm.com/docs-content/v1/content/2a59ee244ed61eb35ab50e45757b42d22f720132/Db2onCloud/images/credential.png)

Figure 2. Create Service Credentials





 Get the Access key and Secret access key from the Credentials 



1. Expand the Credential
2. Note down the access_key_id and secret_access_key



Zoom

![Get Keys](https://cloud.ibm.com/docs-content/v1/content/2a59ee244ed61eb35ab50e45757b42d22f720132/Db2onCloud/images/bucket.png)

Figure 3. Get Access Keys





 Open the Db2 Console to the load data page 



1. Click on Data on the left menu
2. Click on Load Data on the top tab
3. Click on Cloud Object Storage
4.
22 June 2021 

IBM Cloud Object Storage plug-in
: Version 2.1.2 of the IBM Cloud Object Storage plug-in is released. Update your clusters to use the latest version. For more information, see the [IBM Cloud Object Storage plug-in change log](https://cloud.ibm.com/docs/openshift?topic=openshift-cos_plugin_changelog).

Worker node versions
: Worker node fix pack update change log documentation is available for version [4.7.16_1522_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_474716_1522), [4.6.34_1546_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_464634_1546), [4.5.40_1542_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-changelog_archive4540_1542)and [3.11.452_1593_openshift](https://cloud.ibm.com/docs/openshift?topic=openshift-openshift_changelog_311311452_1593).





 21 June 2021 

Ingress change logs
: Updated the [Ingress ALB change log](https://cloud.ibm.com/docs/containers?topic=containers-cluster-add-ons-changelogkube_ingress_changelog) for the release of version 0.47.0 of the Kubernetes Ingress image.

New! The addon options command is now available
: For more information, see [addon options](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_cluster_addon_options).

New! Deploy OpenShift Data Foundation on VPC clusters with a single command
* IBM Cloud Object Storage targets support service-to-service authorizations which eliminates the need to manage keys for your IBM Cloud Object Storage targets.
* The setting feature allows the account to choose the location where the Activity Tracker Event Routing metadata for their account is stored.
* The setting feature allows the account to choose one or more default targets for routing events. This removes the requirement to individually define routes in each region if you want to have all events from all regions to flow to a specific target.
* New /api/v2 APIs have been introduced to replace the existing V1 APIs.
* The V1 APIs are now deprecated and will be removed in a future release.
* For existing Activity Tracker Event Routing users, you will need to follow a migration process to use the new V2 features. For customers that have not yet configured a target using Activity Tracker Event Routing, you will default to V2 and will not require a migration.



Activity Tracker Event Routing available in Sydney (au-syd)
: Activity Tracker Event Routing is now available in the in Sydney (au-syd) region. [Learn more](https://cloud.ibm.com/docs/atracker?topic=atracker-regionsregions-atracker).





 22 March 2022 

Activity Tracker Event Routing available in Frankfurt (eu-de) and London (eu-gb)
: Activity Tracker Event Routing is now available in the Frankfurt (eu-de) and London (eu-gb) regions. [Learn more](https://cloud.ibm.com/docs/atracker?topic=atracker-regionsregions-atracker).





 13 August 2021 

General availability of Activity Tracker Event Routing
: [Learn more about Activity Tracker Event Routing](https://cloud.ibm.com/docs/atracker?topic=atracker-getting-started-routing).
If the output shows Error: failed to untar: a file or directory with the name ibm-object-storage-plugin already exists, delete your ibm-object-storage-plugin directory and rerun the helm fetch command.
8. If you use OS X or a Linux distribution, install the IBM Cloud Object Storage Helm plug-in ibmc. The plug-in automatically retrieves your cluster location and to set the API endpoint for your IBM Cloud Object Storage buckets in your storage classes. If you use Windows as your operating system, continue with the next step.



1. Install the Helm plug-in.

helm plugin install ./ibm-object-storage-plugin/helm-ibmc
2. Verify that the ibmc plug-in is installed successfully.

helm ibmc --help

If the output shows the error Error: fork/exec /home/iksadmin/.helm/plugins/helm-ibmc/ibmc.sh: permission denied, run chmod 755 /Users/<user_name>/Library/helm/plugins/helm-ibmc/ibmc.sh. Then, rerun helm ibmc --help.



9. Optional: Limit the IBM Cloud Object Storage plug-in to access only the Kubernetes secrets that hold your IBM Cloud Object Storage service credentials. By default, the plug-in can access all Kubernetes secrets in your cluster.



1. [Create your IBM Cloud Object Storage service instance](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-cos-understandcreate_cos_service).
2. [Store your IBM Cloud Object Storage service credentials in a Kubernetes secret](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-cos-understandcreate_cos_secret)).
3. From the ibm-object-storage-plugin, navigate to the templates directory and list available files. OS X and Linux

cd templates && ls

Windows

chdir templates && dir
4.
import ibm_boto3
from ibm_botocore.client import Config

api_key = "ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE"
service_instance_id = "85SAMPLE-eDOb-4NOT-bUSE-86nnnb31eaxx"
auth_endpoint = "https://iam.cloud.ibm.com/identity/token"
service_endpoint = "https://s3.us-south.cloud-object-storage.appdomain.cloud"

cos = ibm_boto3.resource('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
endpoint_url=service_endpoint)

response = cos.Bucket('<name-of-bucket>').get_bucket_lifecycle_configuration(
Bucket='string'
)

print("Bucket lifecyle: {0}".format(response))

Show more

var aws = require('ibm-cos-sdk');
var ep = new aws.Endpoint('s3.us-south.cloud-object-storage.appdomain.cloud');
var config = {
endpoint: ep,
apiKeyId: 'ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE',
ibmAuthEndpoint: 'https://iam.cloud.ibm.com/identity/token',
serviceInstanceId: 'crn:v1:bluemix:public:cloud-object-storage:global:a/<CREDENTIAL_ID_AS_GENERATED>:<SERVICE_ID_AS_GENERATED>::',
};
var s3 = new aws.S3(config);

var params = {
Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
cos_service_endpoint = 'https://s3.<region>.cloud-object-storage.appdomain.cloud'

cos = ibm_boto3.client("s3",
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=cos_service_endpoint
)

signedUrl = cos.generate_presigned_url(http_method, Params={
'Bucket': bucket_name, 'Key': key_name}, ExpiresIn=expiration)
print("presigned upload URL =>" + signedUrl)
Show more
Using that same cos resource, all versions of objects could be listed using the following:

versions = s3.Bucket(BUCKET).object_versions.filter(Prefix=key)

for version in versions:
obj = version.get()
print(obj.get('VersionId'), obj.get('ContentLength'), obj.get('LastModified'))

Using a client:

!/usr/bin/env python3

import ibm_boto3
from ibm_botocore.config import Config
from ibm_botocore.exceptions import ClientError

Define constants
API_KEY = os.environ.get('IBMCLOUD_API_KEY')
SERVICE_INSTANCE = os.environ.get('SERVICE_INSTANCE_ID')
ENDPOINT = os.environ.get('ENDPOINT')

BUCKET = "my-versioning-bucket" The bucket that will enable versioning.

Create resource client with configuration info pulled from environment variables.
cosClient = ibm_boto3.client("s3",
ibm_api_key_id=API_KEY,
ibm_service_instance_id=SERVICE_INSTANCE,
config=Config(signature_version="oauth"),
endpoint_url=ENDPOINT
)

response = cosClient.put_bucket_versioning(
Bucket=BUCKET,
VersioningConfiguration={
'Status': 'Enabled'
}
)
Show more

Listing the versions of an object using the same client:

resp = cosClient.list_object_versions(Prefix='some-prefix', Bucket=BUCKET)

Note that the Python APIs are very flexible, and there are many different ways to accomplish the same task.





 Node.js
System.out.printf("Item: %s deleted!n", itemName);
}

}

Show more

import ibm_boto3
from ibm_botocore.client import Config, ClientError

 Constants for IBM COS values
COS_ENDPOINT = "<endpoint>" Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
COS_API_KEY_ID = "<api-key>" eg "W00YiRnLW4a3fTjMB-odB-2ySfTrFBIQQWanc--P3byk"
COS_AUTH_ENDPOINT = "https://iam.cloud.ibm.com/identity/token"
COS_RESOURCE_CRN = "<resource-instance-id>" eg "crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003abfb5d29761c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::"

 Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_RESOURCE_CRN,
ibm_auth_endpoint=COS_AUTH_ENDPOINT,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)

def get_bucket_contents(bucket_name, max_keys):
print("Retrieving bucket contents from: {0}".format(bucket_name))
returnArray = []
try:
files = cos.Bucket(bucket_name).objects.all()
for file in files:
print("Item: {0} ({1} bytes)."
Ensure that your computing environment can allow this process to run.



 Initializing the AsperaTransferManager 

Before initializing the AsperaTransferManager, make sure that you have a working [client](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlclient) (not a resource or session) object.

import ibm_boto3
from ibm_botocore.client import Config
from ibm_s3transfer.aspera.manager import AsperaTransferManager

COS_ENDPOINT = "<endpoint>" Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
COS_API_KEY_ID = "<api-key>"
COS_RESOURCE_CRN = "<resource-instance-id>"
COS_BUCKET_LOCATION = "<location>"

 Create resource
cos = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_RESOURCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)

transfer_manager = AsperaTransferManager(cos)
Show more

You need to provide an IAM API Key for Aspera high-speed transfers. [HMAC Credentials](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-service-credentialsservice-credentials-iam-hmac) are NOT currently supported. For more information on IAM, [click here](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-iam-overview).

To get the highest throughput, split the transfer into a specified number of parallel sessions that send chunks of data whose size is defined by a threshold value.
transfer_mgr.shutdown()

 Constants for IBM COS values
COS_ENDPOINT = "<endpoint>" example: https://s3.us-south.cloud-object-storage.appdomain.cloud
COS_API_KEY_ID = "<api-key>" example: xxxd12V2QHXbjaM99G9tWyYDgF_0gYdlQ8aWALIQxXx4
COS_AUTH_ENDPOINT = "https://iam.cloud.ibm.com/identity/token"
COS_SERVICE_CRN = "<resource-instance-id>" example: crn:v1:bluemix:public:cloud-object-storage:global:a/xx999cd94a0dda86fd8eff3191349999:9999b05b-x999-4917-xxxx-9d5b326a1111::
COS_STORAGE_CLASS = "<storage-class>" example: us-south-standard

 Create client connection
cos_cli = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_SERVICE_CRN,
ibm_auth_endpoint=COS_AUTH_ENDPOINT,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)

 Main Program 
def main():
try:
new_bucket_name = "py.bucket." + get_uuid()
new_text_file_name = "py_file_" + get_uuid() + ".txt"
new_text_file_contents = "This is a test file from Python code sample!!!"
new_large_file_name = "py_large_file_" + get_uuid() + ".bin"
* LocationConstraint is a [valid provisioning code](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-classesclasses-locationconstraint) that corresponds to the endpoint value.









 Code Examples 

Code examples are tested on supported release versions of Python.

In your code, you must remove the angled brackets or any other excess characters that are provided here as illustration.



 Initializing configuration 

This example creates a resource object. A resource provides an object-oriented interface to COS. This allows for a higher level of abstraction than the low-level calls provided by a client object.

Note that some operations (such as Aspera high-speed transfer) require a client object. Aspera itself requires Python version 3.6.

import ibm_boto3
from ibm_botocore.client import Config, ClientError

 Constants for IBM COS values
COS_ENDPOINT = "<endpoint>" Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
COS_API_KEY_ID = "<api-key>" eg "W00YixxxxxxxxxxMB-odB-2ySfTrFBIQQWanc--P3byk"
COS_INSTANCE_CRN = "<service-instance-id>" eg "crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003xxxxxxxxxx1c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::"

 Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
AmazonS3 s3client = createClient( api_key, service_instance_id, endpoint_url, location);

TransferManager transferManager = TransferManagerBuilder.standard()
.withS3Client(s3client)
.withMinimumUploadPartSize(partSize)
.withMultipartCopyThreshold(thresholdSize)
.build();

try {
Upload lrgUpload = transferManager.upload(bucketName, uploadFile.getName(), uploadFile);
lrgUpload.waitForCompletion();
System.out.println("Large file upload complete!");
} catch (SdkClientException e) {
System.out.printf("Upload error: %sn", e.getMessage());
} finally {
transferManager.shutdownNow();
}
}
}

Show more

import os
import uuid
import ibm_boto3
from ibm_botocore.client import Config
from ibm_botocore.exceptions import ClientError
import ibm_s3transfer.manager

def log_done():
print("DONE!n")

def log_client_error(e):
print("CLIENT ERROR: {0}n".format(e))

def log_error(msg):
print("UNKNOWN ERROR: {0}n".format(msg))

def get_uuid():
return str(uuid.uuid4().hex)

def generate_big_random_file(file_name, size):
with open('%s'%file_name, 'wb') as fout:
fout.write(os.urandom(size))

 Retrieve the list of available buckets
def get_buckets():
print("Retrieving list of buckets")
try:
var ep = new aws.Endpoint('s3.us-south.cloud-object-storage.appdomain.cloud');
var config = {
endpoint: ep,
apiKeyId: 'ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE',
ibmAuthEndpoint: 'https://iam.cloud.ibm.com/identity/token',
serviceInstanceId: 'crn:v1:bluemix:public:cloud-object-storage:global:a/<CREDENTIAL_ID_AS_GENERATED>:<SERVICE_ID_AS_GENERATED>::',
};
var s3 = new aws.S3(config);

var params = {
Bucket: 'STRING_VALUE' /* required /
};

s3.getBucketLifecycleConfiguration(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
Show more

import sys
import ibm_boto3
from ibm_botocore.client import Config

api_key = "ZRZDoNoUseOLL7bRO8SAMPLEHPUzUL_-fsampleyYE"
service_instance_id = "85SAMPLE-eDOb-4NOT-bUSE-86nnnb31eaxx"
auth_endpoint = "https://iam.cloud.ibm.com/identity/token"
service_endpoint = "https://s3.us-south.cloud-object-storage.appdomain.cloud"

cos = ibm_boto3.resource('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
access_key = os.environ.get('COS_HMAC_ACCESS_KEY_ID')
secret_key = os.environ.get('COS_HMAC_SECRET_ACCESS_KEY')
 Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
cos_service_endpoint = 'https://s3.<region>.cloud-object-storage.appdomain.cloud'

cos = ibm_boto3.client("s3",
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=cos_service_endpoint
)

signedUrl = cos.generate_presigned_url(http_method, Params={
'Bucket': bucket_name, 'Key': key_name}, ExpiresIn=expiration)
print("presigned download URL =>" + signedUrl)
Show more







 Create a presigned URL to upload an object 

import ibm_boto3
import os

bucket_name = '<bucket name>'
key_name = '<object key name>'
http_method = 'put_object'
expiration = 600 time in seconds, default:600

access_key = os.environ.get('COS_HMAC_ACCESS_KEY_ID')
secret_key = os.environ.get('COS_HMAC_SECRET_ACCESS_KEY')
 Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
cos_service_endpoint = 'https://s3.<region>.cloud-object-storage.appdomain.cloud'

cos = ibm_boto3.client("s3",
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
internet-svcs.challenge-ttl-setting.update Update challenge TTL. 
 internet-svcs.always-use-https-setting.update Enable or disable Always Use HTTPS. 
 internet-svcs.true-client-ip-header-setting.update Enable or disable True client IP header. 
 internet-svcs.image-size-optimization-setting.update Enable or disable image size optimization. 
 internet-svcs.script-load-optimization-setting.update Enable or disable script load optimization. 
 internet-svcs.image-load-optimization-setting.update Enable or disable image load optimization. 
 internet-svcs.minify-setting.update Enable or disable minification for HTML, CSS, or JavaScript files. 
 internet-svcs.min-tls-version-setting.update Change minimum TLS version. 
 internet-svcs.ip-geolocation-setting.update Enable or disable IP geolocation header. 
 internet-svcs.http2-setting.update Enable or disable HTTP2 for the domain. 
 internet-svcs.max-upload-setting.update Change the amount of data that visitors can upload to the website in a single request. 
 internet-svcs.origin-error-page-pass-thru-setting.update Enable or disable the proxy of 502 and 504 error pages that are returned from origin server. 
 internet-svcs.bot-management.update Change Bot Management settings. 







 Viewing events 

Currently, events are available in the Frankfurt region.

IBM Cloud Activity Tracker can have only one instance per location. To view events, you must access the web UI of the IBM Cloud Activity Tracker service in the same location where your service instance is available. For more information, see [Launching the web UI through the IBM Cloud UI](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-launchlaunch_cloud_ui).





 Additional information 

When you monitor IBM Cloud Activity Tracker events that are generated by the IBM Cloud Internet Services, and you identify an API request for which you need additional information, check the requestData field in the event.