vania-janet/multiturn-rag-retrieval-data huggingface dataset:
: Buckets can now be created in a new [Smart Tier storage class](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-billingsmart-tier-pricing-details) that optimizes costs based on usage patterns.





 6 December 2019 

New location!
: Buckets can now be created in a Single Data Center configuration in Singapore. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 15 November 2019 

Lifecycle update
: Data can now be [archived](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-archive) in buckets located in São Paulo, Brazil (sao1).





 24 October 2019 

Compliance update
: [Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutable) is now available for buckets in US Cross Region (us).





 13 October 2019 

New location!
: Buckets can now be created in a Single Data Center configuration in Paris, France. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 11 September 2019 

New feature!
: Changes made to object storage data can be used as an [event source for Cloud Functions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-functions).





 28 August 2019 

New feature!
: Data can be encrypted [using HPCS](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption).





 7 August 2019 

Activity tracking update
* The service ID that you create for an IBM Cloud Object Storage instance is used by IBM Cloud Activity Tracker to authenticate and access the IBM Cloud Object Storage instance.
* You can assign specific access policies to the service ID that restrict permissions on the IBM Cloud Object Storage instance. Restrict the service ID to only have writing permissions on the bucket where you plan to archive the events.
* You can also restrict the IP addresses that are allowed to manage the bucket.



You are responsible for configuring and managing the bucket and the data stored in it.



* If you configure archiving in an EU-managed location, you must configure a bucket that complies with the EU-managed and GDPR regulations.



After you configure archiving,consider the following information:



* Events are automatically archived in a compressed format (.json.gz). Each event preserves its metadata.
* Events are archived within 24-48 hours after you save the configuration.
* Events are archived hourly.
* The first archive file is created when the archiving process runs and there is data.
* Automatic archiving is disabled for an instance when an the credentials that are used to archive data are invalid for over 24 hours.



You have a service plan of 30 days. You configured the instance 10 days ago. You enable archiving today. There will be no archive data for the first 10 days the instance was running.

The archiving process generates multiple files. Each file includes events for the period of time indicated as part of its name. If there is no data, the archive file for that period is empty.



 Archived file format 

The archive directory format looks like this:

year=<YYYY>/month=<MM>/day=<DD>/<accountID>.<YYYY>-<MM>-<DD>.<HHHH>.json.gz

Where

YYYY represents the year; MM represents the month; and DD represents the day.
You can get the encryption details using IBM Cloud UI/CLI. For details, see [Cloud Storage Encryption](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption).
* How can I list all permissions of a bucket?

The IAM feature creates a report at the instance level which may extend to their buckets. It does not specifically report at the bucket level. For details, see [Account Access Report](https://cloud.ibm.com/docs/account?topic=account-access-report).
* How can I monitor bucket changes in the public cloud without using the cloud functions?

You must use cloud functions to get notifications for object changes.
* How can I monitor Object Storage resources?

Use the Activity Tracker service to capture and record Object Storage activities and monitor the activity of your IBM Cloud account. Activity Tracker is used to track how users and applications interact with Object Storage.
* Does an object in a bucket get overwritten if the same object name is used again in the same bucket?

Yes, the object is overwritten.
* How do I get bucket information without using the web console?

Use the Object Storage Resource Configuration API to get bucket information. For details, see [COS configuration](https://cloud.ibm.com/apidocs/cos/cos-configurationreturns-metadata-for-the-specified-bucket) and [COS Integration](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integration).
* How can I manage service credentials for Object Storage instances?

When a service credential is created, the underlying Service ID is granted a role on the entire instance of Object Storage. For details, see [Managing Service credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials).
* Why are parts of my credentials hidden or not viewable?

There may be an issue where the viewer does not have sufficient roles to view the credential information.
mil01 No No No No No No eu-de No Yes eu-de Yes Yes 
 mon01 No No No No No No us-south No Yes us-south Yes Yes 
 par01 No No No No No No eu-de No Yes eu-de Yes Yes 
 sjc01 No No No No No No us-south No Yes us-south Yes Yes 
 sng01 No No No No No No ap-tok No Yes ap-tok Yes Yes 







 Satellite 



 Location Aspera Key Protect (IBM Cloud) Hyper Protect Crypto Services Archive Data Object Lock Immutable Object Storage Activity Tracker Functions Smart Tier Monitoring 

 us-east No Yes No No No No No No No No 
 eu-de No Yes No No No No No No No No 
 eu-gb No Yes No No No No No No No No 
 jp-tok No Yes No No No No No No No No 







 More information 

Learn more about how locations are represented by [endpoints](https://cloud.ibm.com/docs/services/cloud-object-storage?topic=cloud-object-storage-endpoints) for users of IBM Cloud Object Storage.
,
"MaxAgeSeconds": integer
}
...
]
}



* Optional: The REGION where the bucket is present. If this flag is not provided, the program uses the default option that is specified in config.



* Flag: --region REGION



* Optional: Output FORMAT can be only json or text.



* Flag: --output FORMAT











 Put object 



* Action: Upload an object to a bucket in a user's IBM Cloud Object Storage account.
* Usage:ibmcloud cos object-put --bucket BUCKET_NAME --key KEY [--body FILE_PATH] [--cache-control CACHING_DIRECTIVES] [--content-disposition DIRECTIVES] [--content-encoding CONTENT_ENCODING] [--content-language LANGUAGE] [--content-length SIZE] [--content-md5 MD5] [--content-type MIME] [--metadata MAP] [--region REGION] [--output FORMAT]
* Parameters to provide:



* The name of the bucket.



* Flag: --bucket BUCKET_NAME



* The KEY of the object.



* Flag: --key KEY



* Optional: Object data location (FILE_PATH).



* Flag: --body FILE_PATH



* Optional: Specifies CACHING_DIRECTIVES for the request and reply chain.



* Flag: --cache-control CACHING_DIRECTIVES



* Optional: Specifies presentation information (DIRECTIVES).



* Flag: --content-disposition DIRECTIVES



* Optional: Specifies the content encoding (CONTENT_ENCODING) of the object.



* Flag: --content-encoding CONTENT_ENCODING



* Optional: The LANGUAGE the content is in.



* Flag: --content-language LANGUAGE



* Optional: SIZE of the body in bytes. This parameter is useful when the size of the body cannot be determined automatically. (default: 0)



* Flag: --content-length SIZE



* Optional: The base64-encoded 128-bit MD5 digest of the data.



* Flag: --content-md5 MD5
To dynamically provision persistent storage, you must define the type and configuration of the storage that you want.

A [Kubernetes storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) is used to abstract the underlying storage platform that is supported in IBM Cloud so that you don't have to know all the details about supported sizes, IOPS, or retention policies to successfully provision persistent storage in a cluster. IBM Cloud Kubernetes Service provides pre-defined storage classes for every type of storage that is supported. Each storage class is designed to abstract the supported storage tier while giving you the choice to decide on the size, IOPS, and retention policy that you want.

Not finding what you are looking for? You can also create your own customized storage class to provision the type of storage that you want.





 Next steps 

Get started using storage in your cluster by reviewing the following docs.



* [Block Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-block_storage)
* [Block Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-vpc-block)
* [File Storage for Classic for Classic clusters](https://cloud.ibm.com/docs/containers?topic=containers-file_storage)
* [File Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-storage-file-vpc-install)
* [IBM Cloud Object Storage](https://cloud.ibm.com/docs/containers?topic=containers-storage-cos-understand)
* [Portworx](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_about)
* [What benefits are available with the new smart contract lifecycle available on nodes and channels running on Fabric v2.x?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-new-lifecycle)
* [How can I check and interpret the status of my components through the Kubernetes command line?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibm-v2-faq-cr-status)
* [Do you support using certificates from non-IBM Certificate Authorities?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-external-certs)
* [What is the recommended way to manage private keys?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-hsm)
* [Can I integrate my corporate LDAP server with the Certificate Authority (CA) in the IBM Blockchain Platform?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-ldap)
* [What is the process for rotating certificates on a periodic basis?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cert-mgmt)
* [What languages are supported for smart contracts?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-IBP-Overview-1-4)
* [What version of the IBM Blockchain Platform works with the Ansible collection?]
* [Quick start for IBM Cloud Classic File Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile_qs)
* [Deciding on the File Storage for Classic configuration](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile_predefined_storageclass)



* [File storage types](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile-types)
* [IOPS](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile-iops)
* [Reclaim policy](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile-reclaim)
* [Billing type](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile-billing)



* [Adding File Storage for Classic to apps](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storageadd_file)
* [Using existing File Storage for Classic in your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storageexisting_file)



* [Preparing your existing storage](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storageexisting-file-1)
* [Creating a persistent volume and a persistent volume claim](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storageexisting-file-2)



* [Using File Storage for Classic in a stateful set](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storagefile_statefulset)
Yes, IBM COS SDKs are based on the official AWS S3 API SDKs, but are modified to use IBM Cloud features, such as IAM, Key Protect, Immutable Object Storage, and others. When using these SDKs, use HMAC authorization and an explicit endpoint. For details, see [About IBM COS SDKs](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-sdk-about).
* Is there a 100-bucket limit to an account? What happens if I need more?

Yes, 100 is the current bucket limit. Generally, prefixes are a better way to group objects in a bucket, unless the data needs to be in a different region or storage class. For example, to group patient records, you would use one prefix per patient. If this is not a workable solution and you require additional buckets, contact IBM customer support.
* If I want to store my data by using IBM Cloud Object Storage Vault or Cold Vault, do I need to create another account?

No, storage classes (and regions as well) are defined at the bucket level. Simply create a new bucket that is set to the wanted storage class.
* When I create a bucket by using the API, how do I set the storage class?

The storage class (for example, us-smart) is assigned to the LocationConstraint configuration variable for that bucket. This is because of a key difference between the way AWS S3 and IBM Cloud Object Storage handle storage classes. Object Storage sets storage classes at the bucket level, while AWS S3 assigns a storage class to an individual object. For a list of valid provisioning codes for LocationConstraint, see [the Storage Classes guide](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-classes).
* Can the storage class of a bucket be changed? For example, if you have production data in 'standard', can we easily switch it to 'vault' for billing purposes if we are not using it frequently?
* If I set an archive policy on an existing bucket, does the policy apply to existing files?

The policy applies to the new objects uploaded but does not affect existing objects on a bucket. For details, see [Add or manage an archive policy on a bucket](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archivearchive-add).
* Can I unzip a file after I upload it?

A feature to unzip or decompress files is not part of the service. For large data transfer, consider using Aspera high-speed transfer, multi-part uploads, or threads to manage multi-part uploads. See [Store large objects](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-large-objects).
* Can I create a bucket, in the same or different region, with a deleted bucket name?

A bucket name can be reused as soon as 15 minutes after the contents of the bucket have been deleted and the bucket has been deleted. Then, the objects and bucket are irrevocably deleted and can not be restored.

If you do not first empty and then delete the bucket, and instead [delete or schedule the Object Storage service instance for deletion](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-provisiondeleting-a-service-instance), the bucket names will be held in reserve for a [default period of seven (7) days until the account reclamation process](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_commands_resourceibmcloud_resource_reclamations) is completed. Until the reclamation process is complete, it is possible to restore the instance, along with the buckets and objects. After reclamation is complete, all buckets and objects will be irrevocably deleted and can not be restored, although the bucket names will be made available for new buckets to reuse.
How do I allow Aspera High-Speed Transfer through a bucket with context-based restrictions or a firewall? 

The full list (in JSON) of Aspera High-Speed Transfer IP addresses that are used with IBM Cloud Object Storage can be found [using this API endpoint](https://ats.aspera.io/pub/v1/servers/softlayer).





 Does Object Storage provide encryption at rest and in motion? 

Yes. Data at rest is encrypted with automatic provider-side Advanced Encryption Standard (AES) 256-bit encryption and the Secure Hash Algorithm (SHA)-256 hash. Data in motion is secured by using the built-in carrier grade Transport Layer Security/Secure Sockets Layer (TLS/SSL) or SNMPv3 with AES encryption.

If you want more control over encryption, you can make use of IBM Key Protect to manage generated or "bring your own" keying. For details, see [Key-protect COS Integration](https://cloud.ibm.com/docs/key-protect?topic=key-protect-integrate-cos).





 Is there additional encryption processing if a customer wants to encrypt their data? 

Server-side encryption is always on for customer data. Compared to the hashing required in S3 authentication and the erasure coding, encryption is not a significant part of the processing cost of Object Storage.





 Does Object Storage encrypt all data? 

Yes, Object Storage encrypts all data.





 Does Object Storage have FIPS 140-2 compliance for the encryption algorithms? 

Yes, the IBM COS Federal offering is approved for FedRAMP Moderate Security controls, which require a validated FIPS configuration. IBM COS Federal is certified at FIPS 140-2 level 1. For more information on COS Federal offering, [contact us](https://www.ibm.com/cloud/government) via our Federal site.





 Is client-key encryption supported? 

Yes, client-key encryption is supported by using SSE-C, Key Protect, or HPCS.
* [Cleaning up persistent storage](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-file-vpc-managingvpc-storage-remove-file)



[Storage class reference](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-file-vpc-sc-refstorage-file-vpc-sc-ref)





 IBM Cloud storage utilities 

[IBM Cloud storage utilities](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesutilities)



* [Classic: Installing the IBM Cloud Block Storage Attacher plug-in (beta)](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesblock_storage_attacher)



* [Updating the IBM Cloud Block Storage Attacher plug-in](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesupdate_block_attacher)
* [Removing the IBM Cloud Block Volume Attacher plug-in](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesremove_block_attacher)



* [Classic: Manually adding block storage to specific worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesmanual_block)
* [Classic: Attaching raw block storage to non-SDS worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesattach_block)
* [VPC: Adding raw Block Storage for VPC to VPC worker nodes by using the API](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesvpc_api_attach)



* [Detaching raw and unformatted Block Storage for Classic from a worker node in a VPC cluster by using the API](https://cloud.ibm.com/docs/openshift?topic=openshift-utilitiesvpc_api_detach)
* [Storage class reference](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_storageclass_reference)
* [Sample customized storage classes](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_custom_storageclass)



* [Creating topology-aware storage](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile-topology)
* [Specifying the zone for multizone clusters](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_multizone_yaml)
* [Changing the default NFS version](https://cloud.ibm.com/docs/containers?topic=containers-file_storagenfs_version_class)



* [Removing persistent storage from a cluster](https://cloud.ibm.com/docs/containers?topic=containers-file_storagecleanup_file)



* [Understanding your storage removal options](https://cloud.ibm.com/docs/containers?topic=containers-file_storagestorage_delete_options_file)
* [Cleaning up persistent storage](https://cloud.ibm.com/docs/containers?topic=containers-file_storagestorage_remove_file)





[Setting up classic IBM Cloud Block Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-block_storageblock_storage)



* [Quick start for IBM Cloud Block Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-block_storageblock_qs)
* [Installing the IBM Cloud Block Storage for Classic plug-in in your cluster](https://cloud.ibm.com/docs/containers?topic=containers-block_storageinstall_block)



* [Updating the IBM Cloud Block Storage plug-in](https://cloud.ibm.com/docs/containers?topic=containers-block_storageupdate_block)
For more information, see [the account credentials documentation](https://cloud.ibm.com/account?topic=account-service_credentials&interface=uiviewing-credentials-ui).





 Is there a way to enable Key Protect on a Object Storage bucket after the bucket is created? 

No, it is not possible to add Key Protect after creating a bucket. Key Protect can only be added while creating the bucket.





 How can I move data into archive tier? 

You can archive objects using the web console, REST API, and third-party tools that are integrated with IBM Cloud Object Storage. For details, see [COS Archive](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archive).





 Can I use the same Object Storage instance across multiple regions? 

Yes, the Object Storage instance is a global service. Once an instance is created, you can choose the region while creating the bucket.





 Are files scanned for viruses, while being uploaded to COS? 

No, the files are not scanned when uploading to Object Storage. When an object is uploaded, you get upload events by using the Activity Tracker or by using the cloud functions. Then, you can download, scan, and re-upload the object.





 Is it possible to form a Hadoop cluster using Object Storage? 

No, Object Storage is used for the object storage service. For a Hadoop cluster, you need the processing associated with each unit of storage. You may consider the Hadoop-as-a-Service setup.





 Can I generate a "Presigned URL" to download a file and review? 

A PreSigned URL is not generated using the IBM Cloud UI; however, you can use CyberDuck to generate the “pre-signed URL”. It is free. For details, see this [Slack channel](https://ibm-cloudplatform.slack.com/archives/C0VJSU370/p1606859253294800).





 How can I generate a Auth Token using the IAM API Key using REST?
* [getObject](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3.htmlgetObject-com.ibm.cloud.objectstorage.services.s3.model.GetObjectRequest-java.io.File-)









 Delete an item from a bucket 

public static void deleteItem(String bucketName, String itemName) {
System.out.printf("Deleting item: %sn", itemName);
_cos.deleteObject(bucketName, itemName);
System.out.printf("Item: %s deleted!n", itemName);
}

SDK References



* Methods



* [deleteObject](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/AmazonS3.htmldeleteObject-java.lang.String-java.lang.String-)









 Delete multiple items from a bucket 

The delete request can contain a maximum of 1000 keys that you want to delete. While this is very useful in reducing the per-request performance hit, be mindful when deleting a large number of keys. Also take into account the sizes of the objects to ensure suitable performance.

public static void deleteItems(String bucketName) {
DeleteObjectsRequest req = new DeleteObjectsRequest(bucketName);
req.withKeys(
"deletetest/testfile1.txt",
"deletetest/testfile2.txt",
"deletetest/testfile3.txt",
"deletetest/testfile4.txt",
"deletetest/testfile5.txt"
);

DeleteObjectsResult res = _cos.deleteObjects(req);