vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Benefits of using Portworx

 Benefit Description 

 Cloud native storage and data management for stateful apps Portworx aggregates available local storage that is attached to your worker nodes and that can vary in size or type, and creates a unified persistent storage layer for containerized databases or other stateful apps that you want to run in the cluster. By using Kubernetes persistent volume claims (PVC), you can add local persistent storage to your apps to store your data. 
 Highly available data with volume replication Portworx automatically replicates data in your volumes across worker nodes and zones in your cluster so that your data can always be accessed and that your stateful app can be rescheduled to another worker node in case of a worker node failure or reboot. 
 Support to run hyper-converged Portworx can be configured to run [hyper-converged](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/hyperconvergence/) to ensure that your compute resources and the storage are always placed onto the same worker node. When your app must be rescheduled, Portworx moves your app to a worker node where one of your volume replicas resides to ensure local-disk access speed and high performance for your stateful app. 
 Encrypt data with IBM Key Protect You can [set up IBM Key Protect encryption keys](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_encryption) that are secured by FIPS 140-2 Level 2 certified cloud-based hardware security modules (HSMs) to protect the data in your volumes. You can choose between using one encryption key to encrypt all your volumes in a cluster or using one encryption key for each volume. Portworx uses this key to encrypt data at rest and during transit when data is sent to a different worker node. 
 Built-in snapshots and cloud backups You can save the current state of a volume and its data by creating a [Portworx snapshot](https://docs.portworx.com/portworx-install-with-kubernetes/storage-operations/create-snapshots/).
More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 14 December 2018 

New feature!
: Users can use [Immutable Object Storage](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-immutable) to create retention policies that prevent object deletion or modification.

New location!
: Buckets can now be created in a Single Data Center configuration in Mexico City, Mexico. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 12 November 2018 

New location!
: Buckets can now be created in a Single Data Center configuration in Montréal, Canada. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 12 October 2018 

New location!
: Buckets can now be created in a Single Data Center configuration in Seoul, South Korea. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints).





 20 September 2018 

New feature!
: Users can [archive cold data](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-archive) by setting the proper parameters in a bucket lifecycle configuration policy, either using the console, REST API, or a language-specific SDK.





 18 August 2018 

New locations!
: Buckets can now be created in a Single Data Center configuration in Sao Paolo, Brazil and Oslo, Norway.
When you define a retention policy for a bucket, you are specifying that the data is stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner. This policy is enforced until the end of a retention period and the removal of any legal holds. Notice that Immutable Object Storage is available in certain regions only.

[3]: You can use an expiration rule to delete objects automatically after a defined period from the object creation date.

[4]: You can define a retention policy to retain data for a long period of time and have the data available for downloads and queries. If you do not need to query the data, and you just need to keep it for compliance, look into archiving COS files from any of the storage tiers to a long-term offline archive location or use the online Cold Vault option.

[5]: COS provides several options to encrypt your data. By default, all objects that are stored in COS are encrypted by using randomly generated keys and an all-or-nothing-transform (AONT). With COS, you can also manage your keys manually by providing your own encryption keys - referred to as Server-Side Encryption with Customer-Provided Keys (SSE-C). Alternatively, you can choose to use the integration capabilities with IBM Cloud® Key Management Services like IBM® Key Protect and Hyper Protect Crypto Services.

[6]: Resiliency refers to the scope and scale of the geographic area across which your data is distributed. For example, you can choose cross region resiliency to spread your data across several geographical areas, or regional resiliency to spread data across a single region. Notice that a single data center distributes data across devices within a single site only.

Create a custom COS bucket with the storage features and the policies that you identify.[Learn more](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-archiving).

Do not configure a long-term retention policy on a COS bucket if you need access to the data and query it with the Data Engine service.
FAQ 



 Is IBM Cloud® Data Engine transaction safe? 

Data Engine does not support INSERT, UPDATE, or DELETE operations, while it can be used to create Object Storage objects by using the INTO clause. For the created objects in Object Storage, the question arises if this operation is transaction safe and ACID compliant? In other words, if you create an object in Object Storage by using Data Engine, can you assume that this creation is transaction safe? If anything happens before the creation is done, it is rolled back?

The answer is IBM Cloud® Object Storage does not support transactions. However, an object is rolled back if you create it with a single SQL query job on a nonpartitioned object on Object Storage because Object Storage itself does object write in an atomic fashion. If you want to write complex hierarchies of objects with a single SQL job, two scenarios can happen: 1) the job fails half way through and only part of the objects are written, or 2) during job execution some objects are already visible while others are still in process to be written. For the latter scenario, use Hive-style partitioned tables, which means that the SQL job writes a new set of objects into a new Object Storage prefix location. This method does not affect anything in the Hive-style partitioned table. Only when you then also issue an ALTER TABLE … ADD PARTITION with the newly written object location, the data is made available in the Hive-style partitioned table. That ALTER TABLE DDL is in fact also an atomic operation.





 How to handle large time series data sets? 

When you use time-series queries, in general, the more time series that exist, the better the parallelism.

The following example query creates one time series per key, and partitions the data based on the key. Anytime a query is run against "ts", it is run in parallel across time series.

select
key,
ts
from table
using time_series_format(key="key", timetick="time_tick", value="amount") in ts

The issue arises if "ts" is large and thus does not fit in the memory of one machine. To mitigate this issue, modify your key as follows.
Availability and Durability of Block Storage for Classic 

In today's fast-paced economy, companies rely on data in their decision-making. They need secure and immediate access to their data on a moment's notice. Data integrity is high priority because compromised or incomplete data is of no use. Not to mention the dangers that are presented if sensitive data goes missing. When you store your data in IBM Cloud® Block Storage for Classic, it's durable, highly available, and encrypted.



Table 1. Storage durability and availability chart.

 Storage type Purpose Durability Availability Encryption 

 Classic Endurance - <br>0.25 IOPS per GB tier It is designed for workloads with low I/O intensity. These workloads are typically characterized by having a large percentage of data inactive at a time. Example applications include storing mailboxes or departmental level file shares. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>2 IOPS per GB tier It is designed for most general-purpose usage. Example applications include hosting small databases that are backing web applications or virtual machine disk images for a hypervisor. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>4 IOPS per GB tier It is designed for higher-intensity workloads. These workloads are typically characterized by having a high percentage of data active at a time. Example applications include transactional and other performance-sensitive databases. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>10 IOPS per GB tier It is designed for the most demanding workloads such as those created by NoSQL databases, and data processing for Analytics.
Storage options 

You have the option of local storage or SAN (portable SAN) for each virtual server. You can supplement local storage or SAN with other storage products as needed.



 Local storage 

Local storage is built on disks that are local to the virtual server host. Local storage provides improved disk read and write performance. The disks are built in a redundant array of independent disks (RAID) configuration for redundancy, disk replacement, and health monitoring that is fully managed by IBM Cloud®. In newer data centers, this storage is all solid-state drive (SSD) to provide the best performance. For more information about available data centers for local SSD storage, see [Balanced local storage](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-about-virtual-server-profilesbalanced-local-storage).





 Portable SAN storage 

Portable storage volumes are auxiliary storage solutions that are exclusively available on Virtual Servers. The portable SAN is built on IBM Cloud's all flash storage clusters rather than the local host storage. This infrastructure provides greater resiliency if a host fails and can also support much larger volumes. If a host fails, virtual server instances that use SAN-based storage are automatically migrated to other hosts and restarted.

Portable storage is an ideal solution if you want to transfer data between virtual servers that exist in any data center on IBM Cloud's network. Portable storage volumes are useful for database applications that require access to raw, unformatted block-level storage and for moving large data sets between Virtual Servers.



 Portable SAN storage considerations 



* All secondary disks are attached as portable storage. In most cases, these secondary disks can be detached at any time to allow them to be moved to other virtual servers.
* If public virtual servers use balanced local storage, you can't detach primary or secondary disks.
* You can attach the disks to another server if the change doesn't exceed the disk quota or the maximum volume size limit of the targeted virtual server.
* Moved disks are converted to the storage type of the target server.
* When a portable storage volume is attached to a virtual server in a different data center than the original virtual server, IBM Cloud copies the volume to the SAN in the new data center.
A file system and its execution parameters that are used within a container runtime to create a container. The file system consists of a series of layers, which are combined at run time, that are created as the container image is built by successive updates. The container image does not retain state as the container runs.

Container images are stored in a repository that is stored in a namespace.





 Digest 

Digests are used as immutable references to various objects in the registry such as image manifests, layers, and configuration items.

In the context of the registry, an image digest is an immutable reference to an image that identifies an image by using the sha256 hash of the [image manifest](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_manifest). You can use an image digest to ensure that you always reference the same version of an image. Use the long format of the image digest to work with images, such as pulling, pushing, and deleting images.

To find the image digest, run the [ibmcloud cr image-digests](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_digests) command. The [ibmcloud cr image-list](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_list) command also returns the image digest, but, by default, it is in a truncated format. You can add an option to the ibmcloud cr image-list command to return the image digest in the long format.

When you are using the image digest to identify an image, always use the long format.

In Container Registry, any reference to "digest" means "image digest".





 Dockerfile 

A Dockerfile is a text file that contains instructions to build a Docker image.

Typically, a container image is built upon a base image that contains a base operating system, such as Ubuntu. You can incrementally change the base image with your Dockerfile instructions to define the environment that the app needs to run.
First, both maps and reduces are precomputed, so the result of a reduce function is a cheap operation. The operation cost is low even when compared to the significant amounts of IO required to stream hundreds or even thousands of documents from the on-disk storage.

At a deeper level, when a node receives a view request, it asks the nodes that hold the shard replicas of the view's database for the results of the view request from the documents in each shard. As it receives the answers, taking the first answer for each shard replica, the node that services the view request combines the results and streams the final result to the client. As more documents are involved, it takes longer for each replica to stream the results from disk and across the network. The node that services the request also has much more work to do in combining the results from each database shard.

Overall, the goal is for a view request to require the minimum amount of data from each shard. This practice minimizes the time that the data is in transit and being combined to form the final result. Using the power of views to precompute aggregate data is one way to achieve this aim. This practice reduces the time that your application spends waiting for the request to complete.
* How can I de-normalize my data?

In relational databases, normalizing data is often the most efficient way to store data. This practice makes sense when you can use JOIN to easily combine data from multiple tables. You're more likely to need an HTTP GET request for each piece of data with IBM Cloudant. If you reduce the number of requests you need to build a complete picture of a modeled entity, you can present information to your users more quickly.

By using views, you get many of the benefits of normalized data while you maintain the de-normalized version for efficiency.

As an example, in a relational schema, you'd normally represent tags in a separate table and use a connecting table to join tags with their associated documents. This practice allows quick lookup of all documents with a specific tag.

In IBM Cloudant, you'd store tags in a list in each document.
Snapshots 

Snapshots are a feature of IBM Cloud® Block Storage for Classic. A snapshot represents a volume's contents at a particular point in time. With snapshots, you can protect your data with no performance impact and minimal consumption of space. Snapshots are considered your first line of defense for data protection. If a user accidentally modifies or deletes crucial data from a volume, the data can be easily and quickly restored from a snapshot copy.

Block Storage for Classic provides you with two ways to take your snapshots.



* First, through a configurable snapshot schedule that creates and deletes snapshot copies automatically for each storage volume. You can also create extra snapshot schedules, manually delete copies, and manage schedules based on your requirements.
* The second way is to take a manual snapshot.



A snapshot copy is a read-only image of a Block Storage for Classic LUN that captures the state of the volume at a point in time. Snapshot copies are efficient both in the time that is needed to create them and in storage space. A Block Storage for Classic snapshot copy takes only a few seconds to create. It's typically less than 1 second, regardless of the size of the volume or the level of activity on the storage. After a snapshot copy is created, changes to data objects are reflected in updates to the current version of the objects, as if Snapshot copies didn't exist. Meanwhile, the copy of the data remains stable.

A Snapshot copy incurs no performance decrease. Users can easily store up to 50 scheduled snapshots and 50 manual snapshots per Block Storage for Classic volume, all of which are accessible as read-only and online versions of the data.

With snapshots, you can:



* Nondisruptively create point-in-time recovery points,
* Revert volumes to previous points-in-time.



You must purchase some amount of snapshot space for your volume first so you can take snapshots of it. The snapshot space can be added during the initial order or afterward through the Volume Details page. Scheduled and manual snapshots share the snapshot space, so make sure you order enough Snapshot space.
When you define a retention policy for a bucket, you are specifying that the data is stored in a WORM (Write-Once-Read-Many), non-erasable and non-rewritable manner. This policy is enforced until the end of a retention period and the removal of any legal holds. Notice that Immutable Object Storage is available in certain regions only.

[3]: You can use an expiration rule to delete objects automatically after a defined period from the object creation date.

[4]: You can define a retention policy to retain data for a long period of time and have the data available for downloads and queries. If you do not need to query the data, and you just need to keep it for compliance, look into archiving COS files from any of the storage tiers to a long-term offline archive location or use the online Cold Vault option.

[5]: COS provides several options to encrypt your data. By default, all objects that are stored in COS are encrypted by using randomly generated keys and an all-or-nothing-transform (AONT). With COS, you can also manage your keys manually by providing your own encryption keys - referred to as Server-Side Encryption with Customer-Provided Keys (SSE-C). Alternatively, you can choose to use the integration capabilities with IBM Cloud® Key Management Services like IBM® Key Protect and Hyper Protect Crypto Services.

[6]: Resiliency refers to the scope and scale of the geographic area across which your data is distributed. For example, you can choose cross region resiliency to spread your data across several geographical areas, or regional resiliency to spread data across a single region. Notice that a single data center distributes data across devices within a single site only.

Create a custom COS bucket with the storage features and the policies that you identify. [Learn more](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-archiving).

Do not configure a long-term retention policy on a COS bucket if you need access to the data and need to query it with the Data Engine service.
--------------------





 Delete stale data with expiration rules 

Objects that are subject to a bucket's Immutable Object Storage retention policy will have any expiration actions deferred until the retention policy is no longer enforced.

For more about using lifecycle configuration to delete objects, check out the [documentation](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry).

This implementation of the PUT operation uses the lifecycle query parameter to set lifecycle settings for the bucket. This operation allows for a single lifecycle policy definition for a bucket. The policy is defined as a set of rules consisting of the following parameters: ID, Status, Filter, and Expiration.

Not all operations are supported in Satellite environments. For details, see [supported Satellite APIs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-apis-cos-satellite)



 Header Type Description 

 Content-MD5 String Required: The base64 encoded 128-bit MD5 hash of the payload, which is used as an integrity check to ensure that the payload wasn't altered in transit. 



The following snippet shows one way to achieve the content for that particular header.

echo -n (XML block) | openssl dgst -md5 -binary | openssl enc -base64

The body of the request must contain an XML block with the following schema:



 Element Type Children Ancestor Constraint 

 LifecycleConfiguration Container Rule None Limit 1. 
 Rule Container ID, Status, Filter, Expiration LifecycleConfiguration Limit 1000. 
 ID String None Rule Must consist of (a-z,A-Z0-9) and the following symbols:!_.*'()- 
 Filter String Prefix Rule Must contain a Prefix element 
 Prefix String None Filter The rule applies to any objects with keys that match this prefix. 
 Expiration Container Days or Date Rule Limit 1. 
 Days Non-negative integer None Expiration Must be a value greater than 0. 
 Date Date None Expiration Must be in ISO 8601 Format. 



Syntax
Immutable backup network architecture 

In the immutable backup solution architecture, the use of a sandbox is optional. In these solution architectures, a sandbox is defined as an isolated network environment where copies of backups can be accessed or used to start virtual machines (VMs). They are isolated from the production environment so that duplicate IP addresses caused by VMs with the same IP addresses do not clash or recovered VMs interact with production VMs.

While Veeam® DataLabs enables the sandbox concept, it is not yet enabled for VMware NSX-T™ environments as it does not enable the mapping of NSX-T segments to isolated segments. Therefore, these solution architectures enable a sandbox with the use of both Veeam and NSX-T technologies:



* Veeam vPower NFS service - Allows starting of VMs directly from the backup files.
* Veeam data integration API - Enables the mounting of the backup files in a files system.
* NSX-T overlay segments - Enables VMs to be connected to virtual networks abstracted from physical networks.
* NSX-T T1 - Provides routing and gateway firewall capabilities.
* NSX-T distributed firewall - Distributed firewall enables firewall capability on east-west traffic on VMs.



Zoom

![Immutable backup environment network overview](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/vmwaresolutions/images/veeam-cr-sa-sb-ib.svg)

Figure 1. Immutable backup environment network overview

The previous diagram shows an example sandbox that is hosted on the consolidated cluster of the VMware vCenter Server® instance. The customer orders the following resources.



* A vCenter Server instance
* The Veeam service
* A bare metal server that runs Ubuntu 20.04 LTS



When the IaaS is provisioned, the customer:



* Deletes the sample segments that are created during the provisioning process.
* Keeps the workload T0 and T1.