vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Manage the storage class and lifecycle policies for the replicated objects independent of the source, by defining a different storage class and/or lifecycle rules for the target bucket. Similarly, you can store replicas in a bucket in a separate service instance or even IBM Cloud account, and also independently control access to the replicas.







 Getting started with replication 

To get started, here are some prerequisites that must be met:



* Set the the Writer or Manager platform role on the source bucket, or a custom role with the appropriate replication actions (such as cloud-object-storage.bucket.put_replication) assigned.
* You do not need to have access to the target bucket, but do need to have sufficient platform roles to create [new IAM policies](https://cloud.ibm.com/docs/account?topic=account-iamoverviewiamoverview) that allow the source bucket to write to the target bucket.
* Both the source and target buckets must have [versioning enabled](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-versioning).
* The target bucket must not have a legacy bucket firewall enabled, but can use [context-based restrictions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-setting-a-firewall).
* Objects encrypted [using SSE-C](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-sse-c) cannot be replicated, although [managed encryption (SSE-KMS) like Key Protect](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-kp) is fully compatible with replication.
* Objects in an archived state cannot be replicated.



As versioning is a requirement for replication, it is not possible to replicate objects in buckets configured with an [Immutable Object Storage policy](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-immutable).



1.
Emptying a bucket 

This overview focuses on the steps that are needed to access a list of all items in a bucket within an instance of IBM Cloud® Object Storage for the purpose of deleting each one sequentially.

The process of emptying a bucket is familiar to anyone who has to delete buckets in their instance of IBM Cloud Object Storage because a bucket has to be empty to be deleted. There may be other reasons you may wish to delete items, but want to avoid deleting every object individually. This code pattern for the supported SDKs will allow you to define your configuration, create a client, and then connect with that client in order to get a list of all the items in an identified bucket for in order to delete them.

If versioning is enabled, then expiration rules create [delete markers](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-versioning). {important}

It is a best practice to avoid putting credentials in scripts. This example is for testing and educational purposes, and your specific setup should be informed by best practices and [Developer Guidance](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-dev-guide).



 Before you begin 

Specific instructions for downloading and installing SDKs are available for [Python](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-python), [Node.js](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-node), [Java](https://cloud.ibm.com/docs/cloud-object-storage/libraries?topic=cloud-object-storage-java), and [Go](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-using-go).
When a service credential is created, the underlying Service ID is granted a role on the entire instance of Object Storage. For details, see [Managing Service credentials](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials).
* Why are parts of my credentials hidden or not viewable?

There may be an issue where the viewer does not have sufficient roles to view the credential information. For more information, see [the account credentials documentation](https://cloud.ibm.com/account?topic=account-service_credentials&interface=uiviewing-credentials-ui).
* Is there a way to enable Key Protect on a Object Storage bucket after the bucket is created?

No, it is not possible to add Key Protect after creating a bucket. Key Protect can only be added while creating the bucket.
* How can I move data into archive tier?

You can archive objects using the web console, REST API, and third-party tools that are integrated with IBM Cloud Object Storage. For details, see [COS Archive](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archive).
* Can I use the same Object Storage instance across multiple regions?

Yes, the Object Storage instance is a global service. Once an instance is created, you can choose the region while creating the bucket.
* Are files scanned for viruses, while being uploaded to COS?

No, the files are not scanned when uploading to Object Storage. When an object is uploaded, you get upload events by using the Activity Tracker or by using the cloud functions. Then, you can download, scan, and re-upload the object.
* Is it possible to form a Hadoop cluster using Object Storage?

No, Object Storage is used for the object storage service. For a Hadoop cluster, you need the processing associated with each unit of storage. You may consider the Hadoop-as-a-Service setup.
* Can I generate a "Presigned URL" to download a file and review?
IBM Cloud Object Storage plug-in 

View information for updates to the IBM Cloud Object Storage plug-in in your Red Hat® OpenShift® on IBM Cloud® clusters.

Refer to the following tables for a summary of changes for each version of the [IBM Cloud Object Storage plug-in](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-cos-understand).



IBM Cloud Object Storage plug-in versions

 IBM Cloud Object Storage plug-in version Supported? Red Hat OpenShift version support Supported architecture 

 2.2.16 Yes Greater than or equal to 4.6 x86 
 2.2.15 Yes Greater than or equal to 4.6 x86 
 2.2.14 Yes Greater than or equal to 4.6 x86 
 2.2.13 Deprecated Greater than or equal to 4.6 x86 
 2.2.12 Deprecated Greater than or equal to 4.6 x86 





 Change log for version 2.2.16, released 3 July 2023 



* Adds support for auto creation and deletion of user defined bucket names.







 Change log for version 2.2.15, released 19 June 2023 



* Updates UBI image to 8.8-860.
* Updates Golang to 1.19.10.
* [Comparison table for persistent storage options](https://cloud.ibm.com/docs/containers?topic=containers-storage-plan)
* [Main COS page](https://cloud.ibm.com/docs/containers?topic=containers-object_storage)
* [Installing COS](https://cloud.ibm.com/docs/containers?topic=containers-object_storageinstall_cos)
* [Creating COS service instance](https://cloud.ibm.com/docs/containers?topic=containers-object_storagecreate_cos_service)
* [Creating COS secret](https://cloud.ibm.com/docs/containers?topic=containers-object_storagecreate_cos_secret)
* [Decide on the configuration](https://cloud.ibm.com/docs/containers?topic=containers-object_storageconfigure_cos)
* [Create an instance of COS](https://cloud.ibm.com/docs/containers?topic=containers-object_storageadd_cos)
* [Back up and restore information](https://cloud.ibm.com/docs/containers?topic=containers-object_storagecos_backup_restore)
* [Storage Class reference](https://cloud.ibm.com/docs/containers?topic=containers-object_storageconfigure_cos)
Reading and writing Cloud Object Storage 



 Table unique resource identifier 

Different types of table unique resource identifiers exist, depending on the type of target service used. You can either specify Cloud Object Storage locations or database locations. The latter is only supported for target locations of an SQL query, and only for IBM® Db2® on Cloud and IBM® Db2® Warehouse on Cloud database services.



 Cloud Object Storage locations 

This identifier points to a location on Cloud Object Storage with a URI format. The access to this Cloud Object Storage bucket is given through the IBM Cloud® Identity and Access Management (IAM) identity of the user that submitted the SQL statement to Data Engine. So, make sure that the user's IAM identity has the necessary access rights that are granted on the Cloud Object Storage bucket.

The identifier has the following form:

cos://<endpoint>/<bucket>/<path>

Where:

<endpoint> The [endpoint](https://cloud.ibm.com/docs/sql-query?topic=sql-query-readingendpoints) of your Cloud Object Storage instance or its [alias](https://cloud.ibm.com/docs/sql-query?topic=sql-query-readingendpoints).

<bucket> The bucket name:



* For an input URI, the bucket that contains the input object or objects.
* For an output URI, the bucket to which the output objects are to be written.



<path> A more exact specification of the object or objects:



* The specified path is interpreted in a similar way as listing file system contents with ls, interpreting slashes / in object names as a folder hierarchy.



* If the path is identical to the name of an existing (nonempty) object, it matches only that single object.
* If the path is a prefix of multiple objects at a slash / character, it matches all those objects that are not empty. For example, the path mydir/test1 (or mydir/test1/) matches objects mydir/test1/object1, mydir/test1/nested/object2, but not mydir/test100.
.then(() => putObjectLockConfigurationonBucket(newBucketName)) // Put objectlock configuration(i.e. default retention) on COS bucket.
.then(() => getObjectLockConfigurationonBucket(newBucketName)) // Read objectlock configuration on COS bucket.
.then(() => createTextFile(newBucketName, newTextFileName, newTextFileContents)) // Upload an object to cos bucket.
.then(() => putObjectLockRetention(newBucketName, newTextFileName)) // Put objectlock retention(i.e. retain until date) on the object.
.then(() => getObjectLockRetention(newBucketName, newTextFileName)) // Get the configured retention.
.then(() => putObjectLockLegalhold(newBucketName,newTextFileName)) // Put objectlock legalhold on the object.
.then(() => getObjectLockLegalhold(newBucketName,newTextFileName)); // Get the legalhold status.
}
catch(ex) {
logError(ex);
}
}

main();

Show more





 Go 

package main

import (
"bytes"
"fmt"
"time"

"github.com/IBM/ibm-cos-sdk-go/aws"
"github.com/IBM/ibm-cos-sdk-go/aws/credentials/ibmiam"
"github.com/IBM/ibm-cos-sdk-go/aws/session"
"github.com/IBM/ibm-cos-sdk-go/service/s3"
)

const (
apiKey = "<apiKey>"
serviceInstanceID = "<serviceInstanceID>"
cloud-object-storage.object-multipart.delete Abort an incomplete multipart upload of an object 







 Bucket versioning events 

The following table lists the COS versioning events:



Table 8. Versioning events

 Action Description 

 cloud-object-storage.bucket-versioning.create Enable versioning on a bucket 
 cloud-object-storage.bucket-versioning.read Check versioning status of a bucket 
 cloud-object-storage.bucket-versioning.list List versions of objects in a bucket 



For cloud-object-storage.bucket-versioning.create events, the following fields include extra information:



 Field Description 

 requestData.newValue.versioning.state The versioning state of the bucket (enabled or suspended). 
 {: caption="Table 8a. Additional fields for bucket-versioning.create events"} 









 Viewing events 

You can view the Activity Tracker events that are associated with your Object Storage instance by using [IBM Cloud Activity Tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started).

You can only provision 1 instance of the IBM Cloud Activity Tracker service per location.

To view events, you must identify the location where events are collected and available for monitoring. Then, you must access the web UI of the IBM Cloud Activity Tracker instance in that location. For more information, see [Launching the web UI through the IBM Cloud UI](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-observe).



 Management events 

Object Storage global events are forwarded to the IBM Cloud Activity Tracker service instance that is located in Frankfurt.

All other Object Storage management events are forwarded to the IBM Cloud Activity Tracker instance that is associated with the bucket.

To view events, you must access the web UI of the IBM Cloud Activity Tracker instance in the location that is associated with the bucket.





 Data events 

Object Storage data events are forwarded to the IBM Cloud Activity Tracker instance that is associated with the bucket.

To view events, you must access the web UI of the IBM Cloud Activity Tracker instance in the location that is associated with the bucket.







 Analyzing events
* [Classic NFS File Storage for Classic](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-mng-dataencryptvolume)
* [Classic Block Storage](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-mng-data)
* [VPC Block Storage](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-aboutvpc-storage-encryption)
* [IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption)



Depending on the type of storage that you choose, you can set up additional encryption with IBM Key Protect to protect your data in transit and at rest with your own encryption key.



* [VPC Block Storage](https://cloud.ibm.com/docs/containers?topic=containers-vpc-blockvpc-block-encryption)
* [IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption)
* [Portworx](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_encryption)



You can also use an IBM Cloud database service, such as [IBM Cloudant NoSQL DB](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant), to persist data in a managed database outside the cluster. Data that is stored with a cloud database service can be accessed across clusters, zones, and regions. For security-related information, see the database service-specific IBM Cloud documentation.





 Monitoring and logging 

The key to detect malicious attacks in your cluster is the proper monitoring and logging of metrics and all the events that happen in the cluster. Monitoring and logging can also help you understand the cluster capacity and availability of resources for your app so that you can plan accordingly to protect your apps from a downtime.

Does IBM monitor my cluster?
(https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cloud-storage)
* [Do I need multizone region storage for IBM Blockchain Platform nodes?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-cloud-mzr-storage)
* [What versions of Red Hat OpenShift are supported?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-ocp-versions)
* [Is there a trial option available for using a Red Hat OpenShift cluster on IBM Cloud?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-ocp-trial)
* [Is it possible to deploy blockchain nodes to multiple clouds from a single blockchain console?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-multicloud)
* [How can I find what version of the IBM Blockchain Platform that I am running?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-version)
* [How do I get the latest Fabric version and Fabric functionalities on my IBM Blockchain Platform network?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-fabric-upgrade)
* [I am currently using Hyperledger Fabric v1.4.x and want to move to IBM Blockchain Platform for IBM Cloud . Can I continue to use Raft?]
* "description": "Descriptive text about the configuration" - A description of your configuration



The following objects and arrays must be defined to convert, enrich and normalize documents that are uploaded to your collection.



* "conversions": {} - How documents are transformed into JSON that can be enriched.
* "enrichments": [] - Which enrichments are applied to which parts of the JSON.
* "normalizations": [] - Any post enrichment adjustments that are required before the document is stored.



Additionally, the following items are added to the base object by Discovery when the configuration is created/updated:

{
"configuration_id": "4f5b7c7b-ebf4-4963-882e-27eff08f08e3",
"created": "2017-09-13T14:45:03.575Z",
"updated": "2017-09-13T14:45:03.575Z"
}





 Conversion 

Converting documents takes the original source format and using one or more steps transforms it into JSON that can be used for the rest of the ingestion process. Depending on the type of file uploaded, the process is as follows:



* PDF files are converted to HTML using the pdf options, the resulting HTML is then converted to JSON using the html options, and finally the resulting JSON is converted using the json options.
* Microsoft Word files are converted to HTML using the word options, the resulting HTML is then converted to JSON using the html options, and finally the resulting JSON is converted using the json options.
* HTML files are converted to JSON using the html options, and the resulting JSON is converted using the json options.
* JSON files are converted using the json options.



If you configure your collection using [Smart Document Understanding](https://cloud.ibm.com/docs/discovery?topic=discovery-sdu), the PDF and Word conversion settings listed are not used, so changing these conversion settings are ignored.

These options are described in the following sections.
<Key>soldier-bee</Key>
<LastModified>2016-08-25T17:49:06.006Z</LastModified>
<ETag>"37d4c94839ee181a2224d6242176c4b5"</ETag>
<Size>11</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult>
Show more

--------------------





 List objects in a given bucket (deprecated) 

This API is included for backwards compatibility. See [Version 2](https://cloud.ibm.com/docs/cloud-object-storage/api-reference?topic=cloud-object-storage-compatibility-api-bucket-operationscompatibility-api-list-objects-v2) for the recommended method of retrieving objects in a bucket.

Not all operations are supported in Satellite environments. For details, see [supported Satellite APIs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-apis-cos-satellite)

A GET request addressed to a bucket returns a list of objects, limited to 1,000 at a time and returned in non-lexographical order. The StorageClass value that is returned in the response is a default value as storage class operations are not implemented in COS. This operation does not make use of operation specific headers or payload elements.

Syntax

GET https://{endpoint}/{bucket-name} path style
GET https://{bucket-name}.{endpoint} virtual host style





 Optional query parameters 



 Name Type Description 

 prefix String Constrains response to object names beginning with prefix. 
 delimiter String Groups objects between the prefix and the delimiter. 
 encoding-type String If unicode characters that are not supported by XML are used in an object name, this parameter can be set to url to properly encode the response.
(https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-Infrastructure-4-2)
* [What database do the peers use for their ledger?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-v2-IBP-Overview-1-3)
* [What types of off-chain databases are supported with the IBM Blockchain Platform?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-offchain-db)
* [If service discovery is on, will an endorsement request be routed to any peer on the network?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-service-discovery)
* [Do ordering service Raft nodes use Transport Layer Security (TLS) for communication?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-raft-tls)
* [How can I back up and restore components and networks?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-backup-restore)
* [What benefits are available with the new smart contract lifecycle available on nodes and channels running on Fabric v2.x?](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-faqibp-v2-faq-new-lifecycle)
* [How can I check and interpret the status of my components through the Kubernetes command line?]
Setting up classic IBM Cloud File Storage for Classic 

IBM Cloud File Storage for Classic is persistent, fast, and flexible network-attached, NFS-based File Storage for Classic that you can add to your apps by using Kubernetes persistent volumes (PVs). You can choose between predefined storage tiers with GB sizes and IOPS that meet the requirements of your workloads. To find out if IBM Cloud File Storage for Classic is the correct storage option for you, see [Choosing a storage solution](https://cloud.ibm.com/docs/containers?topic=containers-storage-plan). For pricing information, see [Pricing](https://cloud.ibm.com/cloud-storage/file/order).

Classic infrastructure



 Quick start for IBM Cloud Classic File Storage for Classic 

In this quick start guide, you create a 24Gi endurance File Storage for Classic volume in your cluster by creating a PVC to dynamically provision the volume. Then, you create an app deployment that mounts your PVC.

First time using File Storage for Classic in your cluster? Come back here once you are familiar with the [File Storage for Classic](https://cloud.ibm.com/docs/containers?topic=containers-file_storagefile_predefined_storageclass) configurations.



1. Create a file for your PVC and name it pvc.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: silver-pvc
labels:
billingType: hourly
region: Example: us-south
zone: Example: dal13
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 24Gi
storageClassName: ibmc-file-silver
2. Create the PVC in your cluster.

kubectl apply -f pvc.yaml
3. After your silver-pvc PVC is bound, create an app deployment that uses your PVC. Create a file for your deployment and name it deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
* [Decide where to store application files](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-decide_where_to_store_data)



* [Non-persistent data storage](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-11)
* [Learn how to create persistent data storage for your app](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-12)
* [Learn how to move existing data to persistent storage](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-13)
* [Set up backups for persistent storage](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-14)



* [Prepare your code](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-prepare_code)



* [Apply the 12-factor principles](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-16)
* [Store credentials in Kubernetes secrets](https://cloud.ibm.com/docs/Registry?topic=Registry-vm-to-containers-and-kubernetesvm-to-containers-and-kubernetes-17)