vania-janet/multiturn-rag-retrieval-data huggingface dataset:
--------------------





 Temporarily restore an archived object 

This implementation of the POST operation uses the restore query parameter to request temporary restoration of an archived object. The user must first restore an archived object before downloading or modifying the object. When restoring an object, the user must specify a period after which the temporary copy of the object will be deleted. The object maintains the storage class of the bucket.

There can be a delay of up to 12 hours before the restored copy is available for access. A HEAD request can check if the restored copy is available.

To permanently restore the object, the user must copy the restored object to a bucket that does not have an active lifecycle configuration.

Cloud IAM users must have at a minimum the Writer role to restore an object.

Classic Infrastructure users must have at a minimum Write permissions on the bucket and Read permission on the object to restore it.

This operation does not make use of additional operation specific query parameters.



 Header Type Description 

 Content-MD5 string Required: The base64 encoded 128-bit MD5 hash of the payload, used as an integrity check to ensure the payload was not altered in transit. 



The body of the request must contain an XML block with the following schema:



 Element Type Children Ancestor Constraint 

 RestoreRequest Container Days, GlacierJobParameters None None 
 Days Integer None RestoreRequest Specified the lifetime of the temporarily restored object. The minimum number of days that a restored copy of the object can exist is 1. After the restore period has elapsed, temporary copy of the object will be removed. 
 GlacierJobParameters String Tier RestoreRequest None 
 Tier String None GlacierJobParameters Optional, and if left blank will default to the value associated with the storage tier of the policy that was in place when the object was written. If this value is not left blank, it must be set to Bulk if the transition storage class for the bucket's lifecycle policy was set to GLACIER, and must be set to Accelerated if the transition storage class was set to ACCELERATED. 



A successful response returns a 202 if the object is in the archived state and a 200 if the object is already in the restored state.
4000-7999 Gi 300-48000 IOPS 
 8000-9999 Gi 500-48000 IOPS 
 10000-12000 Gi 1000-48000 IOPS 







 Reclaim policy 

Choose if you want to keep your data after the cluster or the persistent volume claim (PVC) is deleted.



* If you want to keep your data, then choose a retain storage class. When you delete the PVC, only the PVC is deleted. The PV, the physical storage device in your IBM Cloud infrastructure account, and your data still exist. To reclaim the storage and use it in your cluster again, you must remove the PV and follow the steps for [using existing File Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storageexisting_file).
* If you want the PV, the data, and your physical File Storage for Classic device to be deleted when you delete the PVC, choose a storage class without retain.







 Billing type 

Choose hourly or monthly. Review the [pricing](https://cloud.ibm.com/cloud-storage/file/order) for more information.

By default, all File Storage for Classic devices are provisioned with an hourly billing type.

If you choose a monthly billing type, when you remove the persistent storage, you still pay the monthly charge for it, even if you used it only for a short amount of time.







 Adding File Storage for Classic to apps 

Create a persistent volume claim (PVC) to [dynamically provision](https://cloud.ibm.com/docs/openshift?topic=openshift-kube_conceptsdynamic_provisioning) File Storage for Classic for your cluster. Dynamic provisioning automatically creates the matching persistent volume (PV) and orders the physical storage device in your IBM Cloud infrastructure account.

Before you begin:
}).promise()
.then(() => {
console.log(Updated metadata for item: ${itemName} from bucket: ${bucketName});
})
})
.catch((e) => {
console.log(ERROR: ${e.code} - ${e.message}n);
});
}
Show more





 Using COPY to update metadata 

function updateMetadataCopy(bucketName, itemName, metaValue) {
console.log(Updating metadata for item: ${itemName});

//set the copy source to itself
var copySource = bucketName + '/' + itemName;

//set the new metadata
var newMetadata = {
newkey: metaValue
};

return cos.copyObject({
Bucket: bucketName,
Key: itemName,
CopySource: copySource,
Metadata: newMetadata,
MetadataDirective: 'REPLACE'
}).promise()
.then((data) => {
console.log(Updated metadata for item: ${itemName} from bucket: ${bucketName});
})
.catch((e) => {
console.log(ERROR: ${e.code} - ${e.message}n);
});
}
Show more







 Using Immutable Object Storage 



 Add a protection configuration to an existing bucket 

Objects written to a protected bucket cannot be deleted until the protection period has expired and all legal holds on the object are removed. The bucket's default retention value is given to an object unless an object specific value is provided when the object is created. Objects in protected buckets that are no longer under retention (retention period has expired and the object does not have any legal holds), when overwritten, will again come under retention. The new retention period can be provided as part of the object overwrite request or the default retention time of the bucket will be given to the object.
The newest images are retained. The age of images is determined by their build date. IMAGECOUNT is the number of images that you want to retain in each repository for the namespace. To return a policy to the default state that keeps all the images, set IMAGECOUNT to All.





 Examples 

Set a policy that retains the newest 20 images within each repository in the namespace birds.

ibmcloud cr retention-policy-set --images 20 birds

Set the policy back to the default state so that you keep all your images in the namespace birds.

ibmcloud cr retention-policy-set --images All birds

For more information about how to use the ibmcloud cr retention-policy-set command, see [Retaining images](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_retention).







 ibmcloud cr retention-run 

Cleans up a namespace by retaining a specified number of images for each repository within a namespace in IBM Cloud Container Registry. All other images in the namespace are deleted.

You can choose whether to exclude all untagged images from the total number of images that you decide to retain.

Where an image, within a repository, is referenced by multiple tags, that image is counted only once. Newest images are retained. Age is determined by when the image was created, not when it was pushed to the registry.

If you want to restore a deleted image, you can list the contents of the trash by running the [ibmcloud cr trash-list](https://cloud.ibm.com/docs/cli?topic=cli-containerregclibx_cr_trash_list) command and restore a selected image by running the [ibmcloud cr image-restore](https://cloud.ibm.com/docs/cli?topic=cli-containerregclibx_cr_image_restore) command.

If an image that you're expecting to see doesn't show in the list that is produced, see [Why doesn't the retention command show all the images?](https://cloud.ibm.com/docs/Registry?topic=Registry-troubleshoot-image-list-retention) for assistance.
ibmc-block-bronze ibm.io/ibmc-block Delete Immediate true 148m
ibmc-block-custom ibm.io/ibmc-block Delete Immediate true 148m
ibmc-block-gold ibm.io/ibmc-block Delete Immediate true 148m
ibmc-block-retain-bronze ibm.io/ibmc-block Retain Immediate true 148m
ibmc-block-retain-custom ibm.io/ibmc-block Retain Immediate true 148m
ibmc-block-retain-gold ibm.io/ibmc-block Retain Immediate true 148m
ibmc-block-retain-silver ibm.io/ibmc-block Retain Immediate true 148m
ibmc-block-silver ibm.io/ibmc-block Delete Immediate true 148m
2. Review the configuration of a storage class.

oc describe storageclass STORAGECLASS

For more information about each storage class, see the [storage class reference](https://cloud.ibm.com/docs/openshift?topic=openshift-block_storageblock_storageclass_reference). If you don't find what you are looking for, consider creating your own customized storage class. To get started, check out the [customized storage class samples](https://cloud.ibm.com/docs/openshift?topic=openshift-block_storageblock_custom_storageclass).
3. Choose the type of block storage that you want to provision.



* Bronze, silver, and gold storage classes: These storage classes provision Endurance storage. With Endurance storage, you can choose the size of the storage in gigabytes at predefined IOPS tiers.
* Custom storage class: This storage class provisions Performance storage. With performance storage, you have more control over the size of the storage and the IOPS.



4. Choose the size and IOPS for your block storage. The size and the number of IOPS define the total number of IOPS (input/ output operations per second) that serves as an indicator for how fast your storage is. The more total IOPS your storage has, the faster it processes read and write operations.
MaximumRetention Container Days ProtectionConfiguration <br><br> * <br><br><br> 
 DefaultRetention Container Days ProtectionConfiguration <br><br> * <br><br><br> 
 Days Integer <br><br> * <br><br><br> MinimumRetention, MaximumRetention, DefaultRetention Valid retention integer 



Example request

PUT /example-bucket?protection= HTTP/1.1
Authorization: {authorization-string}
x-amz-date: 20181011T190354Z
x-amz-content-sha256: 2938f51643d63c864fdbea618fe71b13579570a86f39da2837c922bae68d72df
Content-MD5: GQmpTNpruOyK6YrxHnpj7g==
Content-Type: text/plain
Host: 67.228.254.193
Content-Length: 299

<ProtectionConfiguration>
<Status>Retention</Status>
<MinimumRetention>
<Days>100</Days>
</MinimumRetention>
<MaximumRetention>
<Days>10000</Days>
</MaximumRetention>
<DefaultRetention>
<Days>2555</Days>
</DefaultRetention>
</ProtectionConfiguration>

Example response

HTTP/1.1 200 OK
Date: Wed, 5 Oct 2018 15:39:38 GMT
X-Clv-Request-Id: 7afca6d8-e209-4519-8f2c-1af3f1540b42
Accept-Ranges: bytes
Server: Cleversafe/3.14.1
X-Clv-S3-Version: 2.5
x-amz-request-id: 7afca6d8-e209-4519-8f2c-1af3f1540b42
Content-Length: 0
If you don't include this option, the volumeBindingMode is automatically set to Immediate and the File Storage for Classic instance is created when you create the PVC.

Example for Endurance File Storage for Classic.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibmc-file-bronze-delayed
parameters:
billingType: hourly
classVersion: "2"
iopsPerGB: "2"
sizeRange: '[20-12000]Gi'
type: Endurance
provisioner: ibm.io/ibmc-file
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Example for Performance File Storage for Classic.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ibmc-file-performance-storageclass
labels:
kubernetes.io/cluster-service: "true"
provisioner: ibm.io/ibmc-file
parameters:
billingType: "hourly"
classVersion: "2"
sizeIOPSRange: |-
"[20-39]Gi:[100-1000]"
"[40-79]Gi:[100-2000]"
"[80-99]Gi:[100-4000]"
"[100-499]Gi:[100-6000]"
"[500-999]Gi:[100-10000]"
"[1000-1999]Gi:[100-20000]"
"[2000-2999]Gi:[200-40000]"
"[3000-3999]Gi:[200-48000]"
"[4000-7999]Gi:[300-48000]"
"[8000-9999]Gi:[500-48000]"
"[10000-12000]Gi:[1000-48000]"
type: "Performance"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
Show more
print("Objectlock Retention for {0}=>".format(object_name))
print(response.Retention)

def objectlock_legalhold(bucket_name,object_name):
 Setting the objectlock legalhold status to ON.
cos_cli.put_object_legal_hold(Bucket=bucket_name, Key=object_name, LegalHold={'Status': 'ON'})
 Get objectlock retention of the above object.
response = cos_cli.get_object_legal_hold(Bucket=bucket_name, Key=object_name)
print("Objectlock Legalhold for {0}=>".format(object_name))
print(response.LegalHold)

COS_ENDPOINT = "" Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints -> Ex:https://s3.us-south.cloud-object-storage.appdomain.cloud
COS_API_KEY_ID = "" API Key of the cos instance created Ex: W00YixxxxxxxxxxMB-odB-2ySfTrFBIQQWanc--P3byk
COS_RESOURCE_INSTANCE_CRN = "" API key of cos instance example: xxxd12V2QHXbjaM99G9tWyYDgF_0gYdlQ8aWALIQxXx4

 Create client connection
cos_cli = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT,
ibm_service_instance_id=COS_RESOURCE_INSTANCE_CRN,
ibm_auth_endpoint="https://iam.test.cloud.ibm.com/identity/token"
)
ibmcloud ks cluster rm --cluster $MYCLUSTER -f --force-delete-storage



Depending on the resource it might not be deleted immediately, but retained (by default for 7 days). You can reclaim the resource by deleting it permanently or restore it within the retention period. See this document on how to [use resource reclamation](https://cloud.ibm.com/docs/account?topic=account-resource-reclamation).





 Expand the tutorial 



* Use the [IBM Cloud® Activity Tracker service](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started) to track how applications interact with IBM Cloud services.
* [Add alerts](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-alerts) to your view.
* [Export logs](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-export) to a local file.
* Examine views.py in the sample application and experiment updating the application to capture additional custom metrics. Create an updated image version and update and apply app-log-analysis.yaml to redeploy your updates.







 Related content 



* [Resetting the ingestion key used by a Kubernetes cluster](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-kube_reset_ingestion)
* [Archiving logs to IBM Cloud Object Storage](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-archivingarchiving)
* [Working with monitoring alerts](https://cloud.ibm.com/docs/monitoring?topic=monitoring-alerts)
* [Working with monitoring notification channels](https://cloud.ibm.com/docs/monitoring?topic=monitoring-notificationsnotifications)
For more information, see [Regions](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewregistry_regions).





 Example 

Target the US South region.

ibmcloud cr region-set us-south







 ibmcloud cr retention-policy-list 

List the image retention policies for your account. Image retention policies retain the specified number of images for each repository within a namespace in IBM Cloud Container Registry. All other images in the namespace are deleted. You can also see whether the option to retain all untagged images applies to the policy.

Where an image within a repository is referenced by multiple tags, that image is counted only once. Newest images are retained. Age is determined by when the image was created, not when it was pushed to the registry.

ibmcloud cr retention-policy-list



 Prerequisites 

To find out about the required permissions, see [Access roles for using Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-iamaccess_roles_using).





 Example 

List the retention policies in your account.

ibmcloud cr retention-policy-list

For more information about how to use the ibmcloud cr retention-policy-list command, see [Retaining images](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_retention).







 ibmcloud cr retention-policy-set 

Set a policy to retain the specified number of images for each repository within a namespace in IBM Cloud Container Registry. All other images in the namespace are deleted. When you set a policy it runs interactively, then it runs daily. You can set only one policy in each namespace.

You can choose whether to exclude all untagged images from the total number of images that you decide to retain.

Where an image, within a repository, is referenced by multiple tags, that image is counted only once. Newest images are retained. Age is determined by when the image was created, not when it was pushed to the registry.

If a retention policy deletes an image that you want to keep, you can restore the image.
For more information, see [Creating custom views](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-view_logsview_logs_step7) and [Managing alerts](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-alerts).

For example, you can define a view that reports when an object is archived in a bucket. On that view, you can configure an email alert to notify you when an object has been archived. You might have compliance requirements that require you to control who accesses data that is archived. You can define a view that reports access to a bucket, and define an alert to notify you when that happens.

Define views, dashboard, screens, and alerts in IBM Cloud® Activity Tracker to investigate for abnormal activity and critical actions in your COS bucket, and to comply with regulatory audit requirements.





 Archived file 

Although you define policies to manage your COS objects (logging archived files) at the bucket level, there might be situations where you need to hold on to specific files for longer for compliance and auditing purposes. In COS, you can set different attributes per object that allow you to meet this requirements.



* You can define a legal hold flag to an archive file. Legal holds can be applied to objects during initial uploads or after an object is written. [Learn more](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutableimmutable-terminology-hold).
* You can define an indefinite retention flag to store the object indefinitely until a new retention period is applied. [Learn more](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-immutableimmutable-terminology-indefinite).



Add a legal hold flag or an indefinite retention flag to individual archived files if you need to keep a file longer than the default retention period specified at the bucket level.







 Define the strategy to query archived data 

You can download data locally, and then use your own tools to query the data.
Uploading larger objects in a single PUT request results in the performance limitations of a single thread, and in the event of any failures single-threaded uploads will need to be retried in their entirety (whereas with MPU only the specific part(s) that failed need to be retried). The precise throughput that can be achieved by a single thread varies depending on the network bandwidth from the client to the IBM Cloud endpoint, the rate of packet loss (if any) on that connection, the use of HTTP vs HTTPS, the specific ciphers used in the connection and specific TCP connection parameters (such as window size), as well as other factors. While these factors can be optimized for a single-threaded upload, the optimizations would apply equally to any multi-threaded (multipart) uploads as well.

Personally Identifiable Information (PII): When creating buckets or adding objects, please ensure to not use any information that can identify any user (natural person) by name, location, or any other means.

Not all operations are supported in Satellite environments. For details, see [supported Satellite APIs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-apis-cos-satellite)

Syntax

PUT https://{endpoint}/{bucket-name}/{object-name} path style
PUT https://{bucket-name}.{endpoint}/{object-name} virtual host style



 Optional headers 



 Header Type Description 

 x-amz-tagging string A set of tags to apply to the object, formatted as query parameters ("SomeKey=SomeValue"). 
 x-amz-object-lock-mode string Valid value is COMPLIANCE - required if x-amz-object-lock-retain-until-date is present. 
 x-amz-object-lock-retain-until-date ISO8601 Date and Time Required if x-amz-object-lock-mode is present. 
 x-amz-object-lock-legal-hold string Valid values are ON or OFF. 



Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Table 13. IBM Spectrum Scale storage class reference.

 Storage class name Type Reclaim policy 

 ibm-spectrum-scale-csi-lt Light weight volumes Delete 







 IBM Systems object storage 



Table 14. Storage class reference for IBM Systems storage

 Storage class name Volume binding mode Retain 

 ibm-s3fs-cos Immediate False 
 ibm-s3fs-cos-perf Immediate False 







 IBM VPC block storage 

Review the Satellite storage classes for IBM VPC block storage. You can describe storage classes in the command line with the oc describe sc <storage-class-name> command.



Table 15. Storage class reference for IBM VPC block storage

 Storage class name Default Read IOPS per GB Default Write IOPS per GB Size range (per disk) Hard disk Reclaim policy Volume Binding Mode 

 sat-vpc-block-gold-metroDefault 10 10 10 GB - 4 TB SSD Delete WaitForFirstConsumer 
 ibmc-vpc-block-metro-5iops-tier 5 5 10 GB - 9600 GB SSD Delete WaitForFirstConsumer 
 ibmc-vpc-block-metro-custom Custom Custom Based on IOPS SSD Delete WaitForFirstConsumer 
 ibmc-vpc-block-metro-general-purpose 3 3 10 GB - 16 TB SSD Delete WaitForFirstConsumer 
 ibmc-vpc-block-metro-retain-10iops-tier 10 10 10 GB - 4 TB SSD Retain WaitForFirstConsumer 
 ibmc-vpc-block-metro-retain-5iops-tier 5 5 10 GB - 9600 GB SSD Retain WaitForFirstConsumer 
 ibmc-vpc-block-metro-retain-custom Custom Custom Based on IOPS SSD Retain WaitForFirstConsumer 
 ibmc-vpc-block-metro-retain-general-purpose 3 3 10 GiB - 16 TB SSD Retain WaitForFirstConsumer 







 VMware Storage CSI Driver 

Review the Satellite storage classes for VMware storage. You can describe storage classes in the command line with the oc describe sc <storage-class-name> command.



Table 16.
In addition to the objects that are written when you do a [batch query](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewresult=), the following two objects are also created:



* _checkpoint: Do not delete or modify the objects in this structure as it stores the last offsets of the topic.
* _schema_as_json: Do not delete or modify the objects in this structure as it stores the schema of the Kafka events.



The objects are written to Cloud Object Storage in micro batches. The number of events that are included in one object depends on the specified event size and the provision rate of the topic. The same applies to the times when data is written to Cloud Object Storage. If the provision rate is high, you see more objects within the same timeframe. If it is low, you see a new object at least every 5 minutes.





 Streaming job details 

You can see your active streaming jobs in the Streaming tab of the job list in the UI. The details of a streaming job show that the states differ from batch query processing. Stopping is an extra state that indicates that Data Engine is stopping a running job. And instead of the completed state, a streaming job goes into the stopped state. The streaming states change from queued to running to stopping or failed.

The job details show the following metrics (instead of rows_returned, rows_read and bytes_read):



* last_change_time: Shows when the last object was written to Cloud Object Storage.
* rows_per_second: Shows the number of records that were processed per second in the last micro batch.
* last_activity_time: Shows the time when the streaming job was last active, with or without processing rows.







 Permissions 

The following permissions are needed for creating a stream landing job:



* Permission to create service-to-service authentication.
* Permission to write to IBM Cloud® Secrets Manager or IBM® Key Protect (to store the API key).
* Permission to create service IDs and API keys. The service ID needs to have following permissions:



* Writer role for the Cloud Object Storage bucket.