vania-janet/multiturn-rag-retrieval-data huggingface dataset:
: Buckets created with [Hyper Protect Crypto Services](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hpcs-cr) managed encryption can now use [Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-immutable) to create retention policies that prevent object deletion or modification.





 23 May 2022 

Encryption update
: Buckets created using [Hyper Protect Crypto Services](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hpcs-cr) managed encryption can now use .





 2 May 2022 

New feature!
: It is now possible so manage access [using context-based restrictions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-setting-a-firewall). This offers significant improvements over the existing bucket firewall, and allows for the allowlisting of VPCs and other cloud services, in addition to IP address ranges.





 5 April 2022 

Encryption update
: Buckets can be created in the eu-gb region using [Hyper Protect Crypto Services](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-hpcs-cr) managed encryption.





 9 March 2022 

Encryption update
: New buckets created in eu Cross Region configuration can now use [Key Protect managed encryption](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-kp).





 20 January 2022 

IBM Cloud Satellite
: You can now use your own compute infrastructure to create a Satellite location. Then, you use the capabilities of Satellite to run IBM Cloud services on your infrastructure, and consistently deploy, manage, and control your software workloads. For details, see [IBM Cloud Satellite](https://cloud.ibm.com/docs/satellite)





 15 December 2022 

Encryption update
* For more information about IBM XIV Storage System, see [Platform and application integration for IBM XIV Storage System](https://www.ibm.com/support/knowledgecenter/STJTAG/css/css_kcwelcomepage_xiv.html)







 Private storage in a single-tentant environment 

See the following table to see pros and cons of private storage in a single-tenant environment.



Table 2. Pros and cons of VMware private storage options

 Key Factors or storage options Local Virtual SAN QuantStor NetApp (OnTap Edge) NetApp (NPS) IBM Spectrum Accelerate 

 Type Local SDS SDS SDS Monolithic SDS 
 Performance based on SSD/SA-SCSI specs, further RAID 5 and 10 can be used for read and write gains. 90 K+ IOPS per host that depends on host configuration. 100 VMs per host, 32 hosts per cluster, 3,200 VMs per cluster, only 2,048 protected (v5.5). Up to 20 K IOPS, 200 VMs per host, 64 hosts per cluster, and 6,000 protected VMs per cluster (v6.0). based on types and number of disks that are selected, and RAID configurations and use of iSCSI or NFS. based on types and number of disks that are selected and RAID configure-actions. Depends on model. based on types and number of disks that are selected, RAID configurations, and optional use of SSD disk per hypervisor node. 
 Scalability Limited growth in size and in disk I/O throughput. Virtual machine disk (VMDK) up to 2 TB with v5.5, and up to 62 TB with v6.0. Single QS up to 128 TB (3.x). No scaling up or out. Up to 10 TB; no scaling out. Yes, add disk shelves for capacity and IOPS. Scales from 8 - 325 TB usable space. Min capacity is 3 VMs x 6 drives. Max capacity is 15 VMs x 12 drives. Scales up to 144 virtual arrays and more than 40 PB usable through IBM Hyper-Scale Manager.
Veeam Linux hardened repository 

The key solution component in both the immutable backup and isolated recovery environment solution architectures is the Linux® hardened repository.

To prevent attacks on backups, Veeam® Backup and Replication 11 introduces the Linux hardened repository feature, which makes backups immutable for the required number of days. This action enables protection against threats, such as ransomware and hackers.

Zoom

![Linux hardened repository](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/vmwaresolutions/images/veeam-cr-sa-lhbr.svg)

Figure 1. Linux hardened repository

The Veeam Linux hardened repository feature uses immutability features in the Linux OS. The immutability file attribute is set for each backup file, which assures the contents of the file. Its associated metadata is retained as nonrewriteable and nonerasable until expiration of an applied Immutable Until Date.

Only the following two Veeam services run on the Linux hardened repository server:



* The Veeam Transport Service receives the backup data and instructs the Veeam Immureposvc Service to set the Immutable Until Dates. This service runs as a normal user (nonroot).
* The Veeam Immureposvc Service sets the immutable flag for the backup files, monitors the Immutable Until Dates and removes the immutable flag when it is expired.



A single-use credential is used by the Veeam backup server to initially configure the Linux hardened repository. This credential is not stored by Veeam. During the initial configuration, certificate-based authentication is established between the Veeam backup server and the Veeam Transport Service.

After the initial configuration, Secure Shell (SSH) access is not required for Veeam operations. However, it is required for server updates and other maintenance tasks, unless all console access is done through the server IPMI port.
* Is there any option in Object Storage to enable accelerate data transfer?

Object Storage offers [Aspera](https://www.ibm.com/cloud/object-storage/aspera) service for high speed data transfer.
* How can I compare various attributes of an object in two different buckets?

Use [Rclone](https://rclone.org/commands/rclone_check). It enables you to compare various attributes.
* What is the default retention period for buckets?

There is no default retention period applied. You can set it while creating the bucket.
* Can we add a retention policy to an existing bucket?

Yes, [Retention policies](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutableimmutable-sdk-add-policy) can be added to an existing bucket; however, the retention period can only be extended. It cannot be decreased from the currently configured value.
* Why is there a "legal hold" concept on top of the "retention period"?

A legal hold prevents an object from being overwritten or deleted. However, a legal hold does not have to be associated with a retention period and remains in effect until the legal hold is removed. For details, see [Legal hold and retention period](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutableimmutable-terminology-hold).
* How can I access a private COS endpoint in a data center from another date center?

Use Object Storage [Direct Link Connection](https://cloud.ibm.com/docs/direct-link?topic=direct-link-using-ibm-cloud-direct-link-to-connect-to-ibm-cloud-object-storage) to create a global direct link.
* How does frequency of data access impact the pricing of Object Storage?

Storage cost for Object Storage is determined by the total volume of data stored, the amount of public outbound bandwidth used, and the total number of operational requests processed by the system.
After your bucket is created with Activity Tracker and Monitoring, it may take a few minutes for the rules to take effect.

You are now ready to store data in a secure content store with encryption, monitoring, and audit observability!





 Get started by uploading data 



* See [uploading data](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-upload) for more information.







 Add capabilities 

Add capabilities to protect objects from ransomware and accidental deletion such as [versioning](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-versioning) and [immutable retention polices](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-ol-overview) for supporting immutable storage, and immutable backup and archive data.





 Library of Object Storage tutorials 

Check out the IBM Cloud Tutorials library for more tutorials when deploying solutions with [Cloud Object Storage](https://cloud.ibm.com/docs?tab=tutorials&page=1&pageSize=20&tags=cloud-object-storage).
* [Working with an IBM Spectrum Accelerate system](https://www.ibm.com/support/knowledgecenter/STJTAG/com.ibm.help.xivgen3.doc/MT/UG/xiv_mt_deploying_spectrum_accelerate.html)













 Shared storage options 

IBM Cloud® offers great File Storage options that you can use to connect to VMware in a multi-tenant environment.



 File Storage 

You can order the Bare Metal Servers from the IBM Cloud catalog with ESX. In the IBM Cloud catalog, you can also provision IBM Cloud® File Storage in the same availability zone as your host. For more information about provisioning file storage, see [Ordering and Managing IBM File Storage for IBM Cloud](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-orderingConsoleorderingConsole).

In VMware, three predefined values are provided on Host Device Details Storage – Username, Password (for CHAP authentication), and Host IQN.



 Shared storage in a multi-tenant environment 

See the following table for pros and cons of shared storage in a multi-tenant environment.



Table 3. Pros and cons of VMware shared storage options

 Key factors and storage options File Storage - endurance and performance options 

 Type SDS 
 Performance Predictable storage performance parameters. <br>Endurance option is available in 0.25, 2, 4 or 10 IOPS per GB. <br>Performance option: Client provisions wanted level of performance based on workload needs or price point. <br>Multiple volumes can be striped together to achieve higher IOPS and more throughput. 
 Scalability Volume sizes range 20 - 12 TB. <br>File Share Capacity can be expanded to 12 TB after initial provisioning in GB increments. 
 Protocols NFS 
 Host connections Maximum of 64 for NFS. 
 Use cases Tier 1, 2, and 3 workloads: <br>0.25 IOPS per GB: Low I/O intensity. Example applications include storing mailboxes or department-level file shares. <br>2 IOPS per GB: General purposes.
Planning for storage 

Review the following sections for an overview of the available storage options for your cluster.

Before you can decide what type of storage is the correct solution for your Red Hat® OpenShift® on IBM Cloud® clusters, you must understand the IBM Cloud infrastructure provider, your app requirements, the type of data that you want to store, and how often you want to access this data.

Decide whether your data must be permanently stored.
: Persistent storage: Data stored on persistent storage persists even when the container, the worker node, or the cluster is removed. Use persistent storage in the for stateful apps, core business data or data that must be available due to legal requirements, such as a defined retention period. Persistent storage is also a good option for auditing.
: Non-persistent storage: Your data can be removed when the container, the worker node, or the cluster is removed. Non-persistent storage is typically used for logging information, such as system logs or container logs, development testing, or when you want to access data from the host's file system.

If you must persist your data, analyze if your app requires a specific type of storage. When you use an existing app, the app might be designed to store data in one of the following ways.
: In a file system: The data can be stored as a file in a directory. For example, you could store this file on your local hard disk. Some apps require data to be stored in a specific file system, such as nfs or ext4 to optimize the data store and achieve performance goals.
: In a database: The data must be stored in a database that follows a specific schema. Some apps come with a database interface that you can use to store your data. For example, WordPress is optimized to store data in a MySQL database. In these cases, the type of storage is selected for you.

Determine the type of data that you want to store.
: Structured data: Data that you can store in a relational database where you have a table with columns and rows. Data in tables can be connected by using keys and is usually easy to access due to the pre-defined data model. Examples are phone numbers, account numbers, Social Security numbers, or postal codes.
This allows for lower latency when accessing storage from compute resources co-located within the same data center, or for data requiring a specific geographic location. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints) documentation.





 8 August 2017 

Introducing IBM Cloud Object Storage
: Object Storage is a highly available, durable, and secure platform for storing unstructured data. Unstructured data (sometimes called binary or "blob" data) refers to data that is not highly structured in the manner of a database. Object storage is the most efficient way to store PDFs, media files, database backups, disk images, or even large structured datasets.
fmt.Println(r)
fmt.Println(e)

}

Show more

The typical response is exemplified here.

{
Rules: [{
Filter: {

},
ID: "id3",
Status: "Enabled",
Transitions: {
Days: 5,
StorageClass: "GLACIER"
}]
}]
}





 Immutable Object Storage 

Users can configure buckets with an Immutable Object Storage policy to prevent objects from being modified or deleted for a defined period of time. The retention period can be specified on a per-object basis, or objects can inherit a default retention period set on the bucket. It is also possible to set open-ended and permanent retention periods. Immutable Object Storage meets the rules set forth by the SEC governing record retention, and IBM Cloud administrators are unable to bypass these restrictions.

Note: Immutable Object Storage does not support Aspera transfers via the SDK to upload objects or directories at this stage.

func main() {

// Create Client
sess := session.Must(session.NewSession())
client := s3.New(sess, conf)

// Create a bucket
input := &s3.CreateBucketInput{
Bucket: aws.String("<BUCKET_NAME>"),
}
d, e := client.CreateBucket(input)
fmt.Println(d) // should print an empty bracket
fmt.Println(e) // should print <nil>

// PUT BUCKET PROTECTION CONFIGURATION
pInput := &s3.PutBucketProtectionConfigurationInput{
Bucket: aws.String("<BUCKET_NAME>"),
ProtectionConfiguration: &s3.ProtectionConfiguration{
DefaultRetention: &s3.BucketProtectionDefaultRetention{
Days: aws.Int64(100),
},
For objects greater than 200mb in size, especially in less stable networks or over very long distances where packet loss is a concern, Aspera High-Speed Transfer can deliver excellent performance. Aspera transfers can also upload nested directory structures efficiently within a single request.





 Throttling batch deletes 

The S3 API provides a mechanism for deleting up to 1,000 objects with a single batch delete request. It is recommended to throttle these requests client-side in order to minimize the chances of derogatory performance within the COS System. When the number of deletes issued is too high for the system, the client will receive HTTP 503 errors with an error message indicating "slow down".





 Consistency impacts 

IBM Cloud Object Storage System guarantees immediate consistency for all object operations, which includes object writes, overwrites, deletes, multipart operations, and ACL modifications. Bucket creation is also immediately consistent. Bucket metadata and configuration is eventually consistent, as is the case with other object storage systems, meaning that changes across a highly distributed system may not be synchronized for a short period of time. This occurs as a result of metadata caching that provides significant performance benefits, and also safeguards against the possibility of denial-of-service attacks. Some applications will overwrite the same object, or delete and rewrite the same object repeatedly over a short amount of time. This can cause contention in the indices within the COS System and should be avoided. In the rare case where overwriting data with the same object key (name) at a very high frequency and over extended periods of time is a critical aspect of an application design, a different storage platform (file, block, noSQL, etc) may be a better choice.





 Existence checks 

Applications may want to check if an object exists or has been modified before writing to it. Often this leads to inefficient application logic that will send a HEAD request followed by a PUT or GET request. This anti-pattern results in wasted networking and server resources, and should be discouraged. Instead of using a HEAD request as an existence check within some function, use a conditional request header. These standard HTTP headers will compare MD5 hashes or timestamps to determine whether the data operation should proceed or not.
For example, if you have production data in 'standard', can we easily switch it to 'vault' for billing purposes if we are not using it frequently? 

You can change the storage class by manually moving or copying the data from one bucket to another bucket with the wanted storage class.





 Can the location of a bucket be changed? 

To change a location, create a new bucket in the desired location and move existing data to the new bucket.





 Does data consistency in Object Storage come with a performance impact? 

Consistency with any distributed system comes with a cost, because the efficiency of the IBM Cloud Object Storage dispersed storage system is not trivial, but is lower compared to systems with multiple synchronous copies.





 Aren't there performance implications if my application needs to manipulate large objects? 

For performance optimization, objects can be uploaded and downloaded in multiple parts, in parallel.





 How does IBM Cloud® Object Storage delete expired data? 

Deletion of an object undergoes various stages to prevent data from being accessible (both before and after deletion). For details, see [Data deletion](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-securitysecurity-deletion).





 How many objects can fit in a single bucket? 

There is no practical limit to the number of objects in a single bucket.





 Can I nest buckets inside one another? 

No, buckets cannot be nested. If a greater level of organization is required within a bucket, the use of prefixes is supported: {endpoint}/{bucket-name}/{object-prefix}/{object-name}. The object's key remains the combination {object-prefix}/{object-name}.





 What is the difference between 'Class A' and 'Class B' requests? 

'Class A' requests are operations that involve modification or listing. This includes creating buckets, uploading or copying objects, creating or changing configurations, listing buckets, and listing the contents of buckets.'Class B' requests are those related to retrieving objects or their associated metadata/configurations from the system. There is no charge for deleting buckets or objects from the system.
Locking objects 

Object Lock preserves electronic records and maintains data integrity by ensuring that individual object versions are stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner. This policy is enforced until a specified date or the removal of any legal holds.



 Why use Object Lock? 

Object Lock helps customers govern data preservation and retention requirements by enforcing data immutability for their backup, disaster recovery, and cyber resiliency workloads.

Object Lock ensures that data cannot be deleted by anyone and there is no way to suspend retention on an object. Read the documentation carefully before locking objects with a retention period.

When using Object Lock, it is your responsibility to ensure compliance with any regulations that you (your organization) may be subject to when it comes to preservation and storage of data for long term retention.

When using Object Lock, you are responsible for ensuring that your IBM Cloud Account is kept in good standing per IBM Cloud policies and guidelines for as long as the data is subject to a retention period. Refer to IBM Cloud Service terms for more information.





 Terminology 

There are two ways use Object Lock to protect data: retention periods and legal holds.



* A retention period defines a time frame during which an object is unable to be modified or deleted.
* A legal hold also prevents an object from being altered, but only remains in place until it is explicitly lifted.



It is possible to make use of any combination of these parameters - an object version can have one, both, or neither.



 Retain Until Date (Retention Period) 

If you need to protect an object version for a fixed amount of time, you need to specify a Retain Until Date which determines the period in which it cannot be altered. The object version can be be deleted after this date is passed (assuming there are no legal holds on the object version).

The retention period for new objects can be inherited from the default value set on the bucket, or it can be explicitly defined when writing the object by specifying a Retain Until Date.

When you use bucket default settings, you don’t specify a Retain Until Date. Instead, you specify a duration, in either days or years, for which every object version placed in the bucket should be protected.
Availability and Durability of File Storage for Classic 

In today's fast-paced economy, companies rely on data in their decision-making. They need secure and immediate access to their data on a moment's notice. Data integrity is high priority because compromised or incomplete data is of no use. Not to mention the dangers that are presented by sensitive data if it goes missing. When you store your data in IBM Cloud® File Storage for Classic, it's durable, highly available, and encrypted.



Table 1. Storage durability and availability chart.

 Storage type Use case Durability Availability Encryption 

 Classic Endurance - <br>0.25 IOPS per GB tier It is designed for workloads with low I/O intensity. These workloads are typically characterized by having a large percentage of data inactive at a time. Example applications include storing mailboxes or departmental level file shares. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>2 IOPS per GB tier It is designed for most general-purpose usage. Example applications include hosting small databases that are backing web applications or virtual machine disk images for a hypervisor. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>4 IOPS per GB tier It is designed for higher-intensity workloads. These workloads are typically characterized by having a high percentage of data active at a time. Example applications include transactional and other performance-sensitive databases. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>10 IOPS per GB tier It is designed for the most demanding workloads such as those created by NoSQL databases, and data processing for Analytics.