vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Using Java 

The IBM Cloud® Object Storage SDK for Java provides features to make the most of IBM Cloud Object Storage.

The IBM Cloud Object Storage SDK for Java is comprehensive, with many features and capabilities that exceed the scope and space of this guide. For detailed class and method documentation [see the Javadoc](https://ibm.github.io/ibm-cos-sdk-java/). Source code can be found in the [GitHub repository](https://github.com/ibm/ibm-cos-sdk-java).



 Getting the SDK 

The easiest way to use the IBM Cloud Object Storage Java SDK is to use Maven to manage dependencies. If you aren't familiar with Maven, you can get up and running by using the [Maven in 5-Minutes](https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html) guide.

Maven uses a file that is called pom.xml to specify the libraries (and their versions) needed for a Java project. Here is an example pom.xml file for using the IBM Cloud Object Storage Java SDK to connect to Object Storage.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.cos</groupId>
<artifactId>docs</artifactId>
<version>1.0.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.ibm.cos</groupId>
<artifactId>ibm-cos-java-sdk</artifactId>
Can Object Storage partition the data automatically using HDFS, so I can read the partitions in parallel, for example, with Spark? 

Object Storage supports a ranged GET on the object, so an application can do a distributed striped-read-type operation. Doing the striping is managed by the application.





 Can I restore a bucket from a specific back-up file? 

It is possible to overwrite an existing bucket. Restore options depend on the capabilities provided by the back-up tool you use; check with your back-up provider. As described in [Your responsibilities when using IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-responsibilities), you are responsible for ensuring data back-ups if necessary. IBM Cloud® Object Storage does not provide a back-up service.





 Can a web browser display the content of files stored in IBM Cloud Object Storage? 

Web browsers can display web content in IBM Cloud Object Storage files, using the COS endpoint as the file location. To create a functioning website, however, you need to set up a web environment; for example, elements such as a CNAME record. IBM Cloud Object Storage does not support automatic static website hosting. For information, see [Static websites](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-iam-public-accesspublic-access-static-website) and this [tutorial](https://www.ibm.com/cloud/blog/static-websites-cloud-object-storage-cos).





 If I set an archive policy on an existing bucket, does the policy apply to existing files? 

The policy applies to the new objects uploaded but does not affect existing objects on a bucket. For details, see [Add or manage an archive policy on a bucket](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archivearchive-add).





 Can I unzip a file after I upload it? 

A feature to unzip or decompress files is not part of the service.
* [Bucket](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/Bucket.html)



* Methods



* [listBuckets](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3.htmllistBuckets--)









 List items in a bucket (v2) 

The [AmazonS3](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3.html) object contains an updated method to list the contents ([listObjectsV2](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3.htmllistObjectsV2-com.ibm.cloud.objectstorage.services.s3.model.ListObjectsV2Request-)). This method allows you to limit the number of records that are returned and retrieve the records in batches. This might be useful for paging your results within an application and improve performance.

public static void getBucketContentsV2(String bucketName, int maxKeys) {
System.out.printf("Retrieving bucket contents (V2) from: %sn", bucketName);

boolean moreResults = true;
String nextToken = "";

while (moreResults) {
ListObjectsV2Request request = new ListObjectsV2Request()
.withBucketName(bucketName)
.withMaxKeys(maxKeys)
.withContinuationToken(nextToken);

ListObjectsV2Result result = _cos.listObjectsV2(request);
for(S3ObjectSummary objectSummary : result.getObjectSummaries()) {
FAQ - General 

Frequently asked questions can produce helpful answers and insight into best practices for working with IBM Cloud® Object Storage.



 How can I find out the total size of my bucket by using the API? 

You can use the [Resource Configuration API](https://cloud.ibm.com/apidocs/cos/cos-configurationreturns-metadata-for-the-specified-bucket) to get the bytes used for a given bucket.





 How can I view my buckets? 

You can view and navigate your buckets using the console, CLI or the API.

For example, the CLI command ibmcloud cos buckets will list all buckets associated with the targeted service instance.





 Can I migrate data from AWS S3 into IBM Cloud Object Storage? 

Yes, you can use your existing tools to read and write data into IBM Cloud Object Storage. You need to configure HMAC credentials allow your tools to authenticate. Not all S3-compatible tools are currently unsupported. For details, see [Using HMAC credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main).





 Can I use AWS S3 SDKs with IBM Cloud Object Storage? 

Yes, IBM COS SDKs are based on the official AWS S3 API SDKs, but are modified to use IBM Cloud features, such as IAM, Key Protect, Immutable Object Storage, and others. When using these SDKs, use HMAC authorization and an explicit endpoint. For details, see [About IBM COS SDKs](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-sdk-about).





 Is there a 100-bucket limit to an account? What happens if I need more? 

Yes, 100 is the current bucket limit. Generally, prefixes are a better way to group objects in a bucket, unless the data needs to be in a different region or storage class. For example, to group patient records, you would use one prefix per patient.
...
Volumes:
myvol:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: mypvc
ReadOnly: false
3. Verify that you can write data to your IBM Cloud Object Storage service instance by logging in to the app pod and writing data. Log in to the pod that mounts your PV.

kubectl exec <pod_name> -it bash
4. Navigate to your volume mount path that you defined in your app deployment.
5. Create a text file.

echo "This is a test" > test.txt
6. From the IBM Cloud® dashboard, navigate to your IBM Cloud Object Storage service instance.
7. From the menu, select Buckets.
8. Open your bucket, and verify that you can see the test.txt that you created.







 Using object storage in a stateful set 

If you have a stateful app such as a database, you can create stateful sets that use IBM Cloud Object Storage to store your app's data. Alternatively, you can use an IBM Cloud database-as-a-service, such as IBM Cloudant and store your data in the cloud.

Before you begin:



* [Create and prepare your IBM Cloud Object Storage service instance](https://cloud.ibm.com/docs/containers?topic=containers-storage-cos-understandcreate_cos_service).
* [Create a secret to store your IBM Cloud Object Storage service credentials](https://cloud.ibm.com/docs/containers?topic=containers-storage-cos-understandcreate_cos_secret).
* [Decide on the configuration for your IBM Cloud Object Storage](https://cloud.ibm.com/docs/containers?topic=containers-storage_cos_installconfigure_cos).



To deploy a stateful set that uses object storage:



1. Create a configuration file for your stateful set and the service that you use to expose the stateful set.
Integrated service availability 

The document describes the regions where services and the different kinds of availability that are supported.

For more information about the following services, be sure to check out the respective links:



* [Aspera high-speed transfer](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-aspera)
* [Key Protect (SSE-KP)](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-kp)
* [Hyper Protect Crypto Services](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hpcs)
* [Archive Data](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archive)
* [Object Lock](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-ol-overview)
* [Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-immutable)
* [Activity Tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started)
* [Functions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-functions)
* [Smart Tier](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-billingsmart-tier-pricing-details)
* [Monitoring](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integration)



Downloads that use Aspera high-speed transfer incur extra egress charges. For more information, see the [pricing page](https://www.ibm.com/cloud/object-storage).



 Cross Region
* [IBM Cloud Object Storage header and body information for events delivered to applications](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producersub-header-body-cos)



* [Subscribing to IBM Cloud Object Storage events for a job](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producerobstorage_ev_job)



* [Subscribing to IBM Cloud Object Storage events for a job from the console](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producereventing-cos-job-ui)
* [Subscribing to IBM Cloud Object Storage events for a job with the CLI](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producereventing-cos-job-cli)
* [Viewing event information for a job from the console](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producerview-eventing-cos-job-ui)
* [Viewing event information for a job with the CLI](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producerview-eventing-cos-job-cli)
* [Environment variables for events delivered to jobs](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-producersub-envir-variables-cos)



* [Defining additional event attributes](https://cloud.ibm.com/docs/codeengine?topic=codeengine-eventing-cosevent-produceradditional-attributes-cos)
* [VPC version 4 master and worker node components](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architectureservice-arch-vpc-4)



* [Overview of personal and sensitive data storage and removal options](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architectureibm-data)



* [What information is stored with IBM when using Red Hat OpenShift on IBM Cloud?](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architecturepi-info)
* [How is my information stored and encrypted?](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architecturepi-storage)
* [Where is my information stored?](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architecturepi-location)
* [How can I remove my information?](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architecturepi-removal)
* [Does Red Hat collect information about my cluster?](https://cloud.ibm.com/docs/openshift?topic=openshift-service-architecturepi-rh-telemetry)





[Protecting Red Hat OpenShift on IBM Cloud resources with context-based restrictions](https://cloud.ibm.com/docs/openshift?topic=openshift-cbrcbr)



* [How Red Hat OpenShift on IBM Cloud integrates with context-based restrictions](https://cloud.ibm.com/docs/openshift?topic=openshift-cbrcbr-overview)



* [Protecting Red Hat OpenShift on IBM Cloud resources](https://cloud.ibm.com/docs/openshift?topic=openshift-cbrresources-types-cbr)
* [Copying data to and from pods and containers](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile-cp-data)



* [Storage class reference](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile_storageclass_reference)
* [Sample customized storage classes](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile_custom_storageclass)



* [Creating topology-aware storage](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile-topology)
* [Specifying the zone for multizone clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile_multizone_yaml)
* [Changing the default NFS version](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagenfs_version_class)



* [Removing persistent storage from a cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagecleanup_file)



* [Understanding your storage removal options](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagestorage_delete_options_file)
* [Cleaning up persistent storage](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagestorage_remove_file)





[Setting up classic IBM Cloud Block Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-block_storageblock_storage)



* [Quick start for IBM Cloud Block Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-block_storageblock_qs)
Storage <br><br> * [Block Storage for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-vpc-encryption)<br> * [Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption)<br><br><br> <br><br> * [Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption)<br><br><br> 
 Security <br><br> * Hyper Protect Crypto Services<br><br><br> <br><br> * Hyper Protect Crypto Services<br><br><br> <br><br> * [App ID](https://cloud.ibm.com/docs/appid?topic=appid-mng-dataenable-customer-keys-hpcs)<br><br><br> 
 Logging and monitoring <br><br> * [Activity Tracker Event Routing](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-mng-data)<br> * [Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-mng-data)<br> * [Flow Logs for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-about)<br><br><br> <br><br> * [Activity Tracker Event Routing](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-mng-data)<br> * [Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-mng-data)<br><br><br>
* [Classic File Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-classic-file)
* [Classic Block Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-classic-block)
* [File Storage for VPC](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-vpc-file)
* [Block Storage for VPC](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-vpc-block)



* [Persistent storage options for multizone clusters](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-multizone)



* [IBM Cloud Object Storage](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-cos)
* [Portworx](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-portworx)
* [Databases](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-databases)
* [OpenShift Data Foundation](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-planstorage-plan-odf)





[Understanding Kubernetes storage basics](https://cloud.ibm.com/docs/openshift?topic=openshift-kube_conceptskube_concepts)



* [Persistent volumes and persistent volume claims](https://cloud.ibm.com/docs/openshift?topic=openshift-kube_conceptspvc_pv)
How do I access the reclaimed resources? 

Create a new set of credentials to access the restored resources.





 Can I host a website using a Object Storage bucket? 

You can use Object Storage bucket to host a static website. For details, see [Hosting Website using COS](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-static-website-tutorial).





 Are REST and cURL commands supported for Object Storage bucket creation using HMAC credentials? 

Yes, you should setup an authorization header. For details, see [Using HMAC Signature](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hmac-signature).





 What kind of IAM authorization is required to edit a bucket's authorized IPs list? 

You must have 'Manager' privilege on the bucket to manage the firewall and to set the authorizations.





 Can I convert a single region Object Storage bucket to cross region without having to copy objects? 

No, you must copy objects to the target bucket. For details, see [COS Region Copy](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-region-copy).





 Is there a way to verify an object’s integrity during an upload to Object Storage? 

Object Storage supports object integrity and ensures that the payload is not altered during transit.





 How can I set a notification when usage in a Object Storage instance is near a certain billing amount? 

You can use a "soft" bucket quota feature by integrating with Metrics Monitoring and configuring for notifications. For details on establishing a hard quota that prevents usage beyond a set bucket size, see [Using Bucket Quota](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-quota).





 Why am I unable to delete a Object Storage instance? 

It isn't possible to delete an instance if the API key or Service ID being used is locked. You'll need to navigate in the console to Manage > Access (IAM) and unlock the API Key or Service ID.
Versioning objects 

Versioning allows multiple revisions of a single object to exist in the same bucket. Each version of an object can be queried, read, restored from an archived state, or deleted. Enabling versioning on a bucket can mitigate data loss from user error or inadvertent deletion. When an object is overwritten, a new version is created, and the previous version of the object is automatically preserved. Therefore, in a versioning-enabled bucket, objects that are deleted as a result of accidental deletion or overwrite can easily be recovered by restoring a previous version of the object. If an object is deleted, it is replaced by a delete marker and the previous version is saved (nothing is permanently deleted). To permanently delete individual versions of an object, a delete request must specify a version ID. A GET request for an object will retrieve the most recently stored version. If the current version is a delete marker, IBM COS returns a 404 Not Found error.

After a bucket has enabled versioning, all the objects in the bucket are versioned. All new objects (created after enabling versioning on a bucket) will receive a permanently assigned version ID. Objects created before versioning was enabled (on the bucket) are assigned a version of null. When an object with a null version ID is overwritten or deleted it is assigned a new version ID. Suspending versioning does not alter any existing objects, but will change the way future requests are handled by IBM COS. Once enabled, versioning can only be suspended, and not fully disabled. Therefore, a bucket can have three states related to versioning: 1. Default (unversioned), 2. Enabled, or 3. Suspended.



 Getting started with versioning 

First, create a new bucket with object versioning enabled.



1. After navigating to your object storage instance, click on Create bucket.
2. Choose a region and resiliency, then look for Object versioning and toggle the selector to Enabled.



Zoom

!
: In addition to [usage metrics](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integrationmm-cos-metrics-usage), IBM Cloud Monitoring can now track [request metrics](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integrationmm-cos-metrics-usage) for buckets.





 31 March 2021 

New feature!
: Buckets can now be [configured to version objects](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-versioning), allowing for non-destructive overwrites and deletes.

Compliance update
: Rules set in the [Security and Compliance Center](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-manage-security-compliance) can now be enforced using the disallow action.

New feature!
: Buckets can now be [configured to have a hard quota](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-quota) to control costs by limiting the maximum amount of storage available for that bucket.





 15 March 2021 

Activity tracking update
: Updates to a bucket's metadata using the [Resource Configuration API](https://cloud.ibm.com/apidocs/cos/cos-configuration) (such as [adding or modifying a firewall](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-setting-a-firewall)) will now show the details of the change in the requestData fields shown in [Activity Tracker](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-at).





 15 December 2020 

New feature!
* [Upload](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/transfer/Upload.html)



* Methods



* [shutdownNow](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/transfer/TransferManager.htmlshutdownNow--)
* [upload](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/transfer/TransferManager.htmlupload-java.lang.String-java.lang.String-java.io.File-)
* [waitForCompletion](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/transfer/internal/AbstractTransfer.htmlwaitForCompletion--)









 Using Key Protect 

Key Protect can be added to a storage bucket to encrypt sensitive data at rest in the cloud.



 Before You Begin 

The following items are necessary in order to create a bucket with Key-Protect enabled:



* A Key Protect service [provisioned](https://cloud.ibm.com/docs/key-protect?topic=key-protect-provision)
* A Root key available (either [generated](https://cloud.ibm.com/docs/key-protect?topic=key-protect-create-root-keys) or [imported](https://cloud.ibm.com/docs/key-protect?topic=key-protect-import-root-keys))







 Retrieving the Root Key CRN 



1. Retrieve the [instance ID](https://cloud.ibm.com/docs/key-protect?topic=key-protect-retrieve-instance-ID) for your Key Protect service
2.