vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Step 3: Adding external storage to your server 

External storage can be added to your provisioned server, or servers. You can use the external storage as a backup device, or as a snapshot to quickly restore your database in a test environment. In the example, block storage is used for both archiving log files of the database and online and offline backups for the database. The fastest block storage (10 IOPS per GB) was selected to help assure a minimum backup time. Slower block storage might be sufficient for your needs. For more information about IBM Cloud® Block Storage for Classic, see [Getting started with Block Storage](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-startedgetting-started).

IBM Cloud storage LUNS can be provisioned with two options - Endurance and Performance. Endurance tiers feature pre-defined performance levels and other features, such as [snapshot](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-snapshots) and replication. A custom Performance environment is built with allocated input/output operations per second (IOPS) in the range 100 - 1,000.



 Setting up external storage 



1. Log in to the [IBM Cloud console](https://cloud.ibm.com/) with your unique credentials.
2. Expand the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/icons/icon.hamburger.svg) and select Classic Infrastructure.
3. Select Storage > Block Storage > Order Block Storage.
4. Select the specifics for your storage needs. Table 1 contains recommended values, including 10 IOPS/GB for a demanding database workload.





Table 3. Recommended values for block storage

 Field Value 

 Location US South, DAL10 
 Billing Method Monthly (default) 
 Size 1000 GB 
 Endurance (IOPS tiers) 10 IOPS/GB 
 Snapshot space 0 GB 
 OS Type Windows 2008+ 





5. Review the Order Summary.
6.
* UI
* CLI
* API
* Terraform







 Adjusting IOPS 

With this feature, IBM Cloud® Block Storage for Classic storage users can adjust the IOPS of their existing Block Storage for Classic immediately. They don't need to create a duplicate or manually copy data to new storage. Users don't experience any kind of outage or lack of access to the storage while the adjustment is taking place.

Billing for the storage is updated to add the pro-rated difference of the new price to the current billing cycle. The full new amount is billed in the next billing cycle.



 Advantages of Adjustable IOPS 



* Cost management – Some clients might need high IOPS only during peak usage times. For example, a large retail store has peak usage during the holidays and might need higher IOPS rate on their storage then. However, they don't need higher IOPS in the middle of the summer. This feature allows them to manage their costs and pay for higher IOPS when they need it.







 Limitations 

Clients can't switch between Endurance and Performance when they adjust their IOPS. However, they can specify a new IOPS tier or IOPS level for their storage based on the following criteria and restrictions:



* If original volume is Endurance 0.25 tier, IOPS tier can’t be updated.
* If original volume is Performance with less than or equal to 0.30 IOPS/GB, options available include only the size and IOPS combinations that result in less than or equal to 0.30 IOPS/GB.
* If original volume is Performance with more than 0.30 IOPS/GB, options available include only the size and IOPS combinations that result in more than 0.30 IOPS/GB.





Table 1. Available IOPS based on volume size for Performance volumes

 Volume size (GB) IOPS range 

 10 - 39 100 - 1,000 
 40 - 79 100 - 2,000 
 80 - 99 100 - 4,000 
 100 - 499 100 - 6,000 
 500 - 999 100 - 10,000 
 1,000 - 1,999 100 - 20,000 
 2,000 - 2,999 200 - 40,000 
 3,000 - 3,999 200 - 48,000
* Canonicalization can be a complicated procedure with many corner cases and ambiguities that significantly increase the complexity of an API. The additional complexity of transforming noncanonical input makes the service harder to develop and maintain. It also expands the testing and attack surfaces.
* If canonicalization decisions are tied to specific libraries used by the service, it could be significantly harder to transition a service implementation to a new technology.
* If canonicalization is done lazily[1], the burden falls on clients to compute equivalency of values. Also, collection sorting behavior might [surprise](https://en.wikipedia.org/wiki/Principle_of_least_astonishment) some client developers regardless of how order is computed for noncanonical values.
* If canonicalization is done eagerly[2], some clients — [declarative orchestration engines](https://en.wikipedia.org/wiki/Infrastructure_as_code) in particular — could experience undesired behavior if a value the client sets is silently transformed. The client might erroneously assume that a canonicalizing transformation it observes is a conflicting update (from another client in a race) and repeatedly attempt to reset the value.
* Besides increasing overall attack surface, certain classes of attacks specifically depend on bugs where different code paths disagree on canonicalization logic.





 Exceptions 

If an industry-standard format used has noncanonical forms that a client developer would reasonably expect to be able to send, and canonicalization is well-documented and widely understood, eager canonicalization SHOULD be used.

For example, [RFC 3339 date-times values](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesdate-time-accepted-formats) with timezone offsets SHOULD be canonicalized and returned in UTC. The ubiquitous support for such canonicalization mitigates the downsides, and together with a developer's expectation that a full industry-standard format would be supported, the benefits exceed the risks in such a case.







 Sanitation and validation 

Definitively invalid or ambiguous input MUST NOT be ignored and therefore MUST result in an error. Validation is better than sanitation.

There are several reasons ignoring invalid input is dangerous and ill-advised. The following are hazards of a service that ignores invalid input:
However, that is where the similarities end.

Replication keeps your data in sync in two different locations. Only one of the volume pair (primary volume and replica volume) can be active at a time. The replication process automatically copies information from the active volume to the inactive volume based on the replication schedule. For more information about replica volumes, see [Replicating data](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-replication).

Duplication creates a copy of your volume based on a snapshot in the same availability zone as the parent volume. The duplicate volume inherits the capacity and performance options of the original volume by default and has a copy of the data up to the point-in-time of a snapshot. The duplicate volume can be dependent or independent from the original volume, and it can be manually refreshed with data from the parent volume. You can adjust the IOPS or increase the volume size of the duplicate without any effect on the parent volume.



* A dependent duplicate volume does not go through the conversion of becoming independent, and can be refreshed at any time after it is created. The system locks the original snapshot so that snapshot cannot be deleted while the dependent duplicate exists. The parent volume cannot be canceled while the dependent duplicate volume exists. If you want to cancel the parent volume, you must either cancel the dependent duplicate first or convert it to an independent duplicate.
* An independent duplicate is superior to the dependent duplicate in most regards, but it cannot be refreshed immediately after creation because of the lengthy conversion process. It can take up to several hours based on the size of the volume. For example, it might take up to a day for a 12-TB volume. However, after the separation process is complete, the data can be manually refreshed by using another snapshot of the original parent volume.



For more information about duplicates, see [Creating and managing duplicate volumes](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-duplicatevolume).



Table 1. Comparison of features between different types of volume copies.
This table has row and column headers. The row headers identify the capability.
There should be no adverse interactions when using Object Lock with other Object Storage features, such as setting CORS policies, setting IP firewalls or condition based restrictions, bucket quotas, or Code Engine.







 IAM actions 

There are new IAM actions associated with Object Lock.



 IAM Action Role 

 cloud-object-storage.bucket.get_object_lock_configuration Manager, Writer, Reader 
 cloud-object-storage.bucket.put_object_lock_configuration Manager, Writer 
 cloud-object-storage.object.get_object_lock_retention Manager, Writer, Reader 
 cloud-object-storage.object.put_object_lock_retention Manager, Writer 
 cloud-object-storage.object.get_object_lock_retention_version Manager, Writer, Reader 
 cloud-object-storage.object.put_object_lock_retention_version Manager, Writer 
 cloud-object-storage.object.get_object_lock_legal_hold Manager, Writer, Reader 
 cloud-object-storage.object.put_object_lock_legal_hold Manager, Writer 
 cloud-object-storage.object.get_object_lock_legal_hold_version Manager, Writer, Reader 
 cloud-object-storage.object.put_object_lock_legal_hold_version Manager, Writer 



Be advised that users with the Writer role are capable of making objects un-deletable for many years (possibly thousand of years). Be careful, and consider crafting custom roles that do not allow most users to set a Retain Until Date.





 Activity Tracker events 

Object Lock generates additional events.



* cloud-object-storage.bucket-object-lock.create
* cloud-object-storage.bucket-object-lock.read
* cloud-object-storage.object-object-lock-legal-hold.create
* cloud-object-storage.object-object-lock-legal-hold.read
* cloud-object-storage.object-object-lock-retention.create
* cloud-object-storage.object-object-lock-retention.read



For cloud-object-storage.bucket-object-lock.create events, the following fields provide extra information:



 Field Description
The service includes a threshold for object size and storage period consistent with the intended use of this service: storing cold, inactive data.

[10]: Flex is used for dynamic workloads where access patterns are more difficult to predict. Depending on usage, if the cost of cooler storage that is combined with retrieval charges exceeds a cap value, then the storage charge increases and no any retrieval charges apply. If the data isn't accessed frequently, Flex storage can be more cost effective than Standard storage. If cooler usage patterns become more active, Flex storage is more cost effective than Vault or Cold Vault storage. No threshold object size or storage period applies to Flex buckets.

Objects that are subject to a bucket's Immutable Object Storage retention policy will have expiration actions deferred until the retention policy is no longer enforced.





 Encrypt data with your own key 

By default, all objects that are stored in COS are encrypted by using randomly generated keys and an all-or-nothing-transform (AONT). When you archive data into your COS instance, your objects are automatically encrypted with data encryption keys (DEKs). When you need to access a bucket, the service checks your user permissions and decrypts the objects within the bucket for you. This encryption model is called provider-managed encryption.

While this default encryption model provides at-rest security, some workloads need full control over the data encryption keys used.

COS provides several options to encrypt your data. This encryption model is called customer-managed encryption:



* You can manage your keys manually by providing your own encryption keys - referred to as Server-Side Encryption with Customer-Provided Keys (SSE-C).
* You can choose to use the integration capabilities with IBM Cloud® Key Management Services like IBM® Key Protect and Hyper Protect Crypto Services.

You can use IBM® Key Protect to provision encrypted keys for apps across IBM Cloud services. As you manage the lifecycle of your keys, you can benefit from knowing that your keys are secured by FIPS 140-2 Level 3 certified cloud-based hardware security modules (HSMs) that protect against the theft of information.
Index management 

Index management, also referred to as data skipping, can significantly boost performance and reduce cost of SQL queries by skipping over irrelevant data. Data skipping indexes apply to structured data sets in IBM Cloud® Object Storage and store summary metadata for each object in the data set. The indexes are stored in Cloud Object Storage in a user-provided bucket, similarly to the data. SQL queries benefit from an index by skipping over all objects whose metadata indicates that they are not relevant to the indicated query.



 Benefits 



* Boosts performance of your queries
* Reduces cost of your queries







 Overview 

For each of the columns in an object, summary metadata can include minimum and maximum values, a list or bloom filter of the appearing values, or other metadata that represents the data in that column. The summary metadata is then used during query evaluation to skip over objects that do not contain any relevant data. All formats are supported, including Parquet, ORC, CSV, and JSON. Data skipping is used for performance optimization. Data skipping does not affect the content of query results. IBM Cloud® Data Engine currently supports metaindexes only. Metaindexes are indexes on a higher level, thus they index objects instead of rows.

As Data Engine charges on a per-query basis based on the amount of data scanned, reducing the number of bytes scanned per query, reduces cost while it improves performance. For data skipping to work well and for good performance overall, use the [best practices for data layout](https://www.ibm.com/cloud/blog/big-data-layout), such as using the Parquet format and adopting Hive-style partitioning. Ideally, create tables by using the [Cloud Object Storage catalog](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-hivemetastore). Data skipping complements these best practices and provides significant cost savings and performance benefits.

To use this feature, you must create indexes on one or more columns of the data set. Start by indexing columns that you query most often in the WHERE clause. A full stop (".") in any column name is not supported.

The following four index types are supported:
Use ValueList for a column if the number of distinct values for that column per object is typically much smaller than the total number of values for that column per object (otherwise, the index can be undesirably large). Use BloomFilter if the number of distinct column values per object is high.

Indexes, or data skipping metadata, are stored in a location you specify. Metadata is typically much smaller than the data itself. If changes are made to some of the objects in the data set after index creation, refresh the indexes. Otherwise, data skipping still works correctly, but it cannot skip the changed objects.





 Usage 



 Sample data 

The sample data set used in this documentation originates from the meter_gen[data generator](https://github.com/gridpocket/project-iostack/tree/master/meter_gen) that was developed by [GridPocket](https://www.gridpocket.com/en/) in the context of the [IOStack project](http://iostack.eu/). It generates electricity, water, and gas meter readings, along with their associated timestamps, geospatial locations, and additional information. The data set is in Parquet format, has 9 GB, and is publicly available to use with Data Engine at cos:\us-geosqlmetergen. The queries that are listed in the examples are also available in the UI under Samples > Index management. Examples for queries when you work on tables are available under Samples > Index management on tables. In the UI samples, under Basic index creation, you find a fast running index creation example that creates only MinMax indexes. Advanced index creation is an example for using all index types and it takes longer to finish.





 Assigning a base location for data skipping indexes 

Indexes are stored in Cloud Object Storage in a bucket of your choice. Before you start to create indexes, first configure the default location, called base location, for indexes created in your Data Engine instance. When you use Hive tables, you can also configure the index location per Hive table.

To assign your base location, use the following command:
Document updates
: Improve reliability of document updates during heavy load.
: Return a 400 response for new_edits=false document update without revisions.

Replicator
: Introduce numerous performance improvements.







 June 2022 



 10 June 2022 

The following changes were made in build 8310:

All databases
: Fix skip and limit parameters for _all_dbs and _dbs_info endpoints.

Attachments
: Wait until attachments are uploaded before response to client.

Bulk docs
: Return 500 errors if timeouts occur when documents are bulk loaded.

Compaction
: Retain compactor state across node reboots.

IAM auth
: Retry authz endpoint on failure.

Indexes
: Prevent creation of indexes with empty name or ddoc fields.
: Improve index building during shard splitting.

Replicator
: Improve replicator stability during timeouts.







 May 2022 



 25 May 2022 

Virtual Private Endpoints (VPE)
: VPE can now be created for IBM Cloudant instances on dedicated hardware.





 13 May 2022 

The following changes were made in build 8299:

Runtime environment
: Upgrade runtime environment to Erlang/OTP 23 with fix for memory leak.

Replicator
: Fix function_clause error for replicated changes with a target VDU.







 April 2022 



 14 April 2022 

The following changes were made in build 8287:

Runtime environment
: Downgrade runtime environment to Erlang/OTP 20.







 March 2022 



 24 March 2022 

The following changes were made in build 8278:

IAM auth
: Improve compatibility during upgrades.

Document updates and compaction
: Improve efficiency of updating and compacting conflicted documents.





 11 March 2022 

The following changes were made in build 8269:

Runtime environment
: Upgrade runtime environment to Erlang/OTP 23.





 04 March 2022 

The following changes were made in build 8266:

Bulk docs
Data-intensive bare metal with SDS, me4c.4x32.1.9tb.ssd: If you need extra local storage for performance, use this disk-heavy flavor that supports software-defined storage (SDS). This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 4 / 32 GB 2 TB HDD / 960 GB SSD 1.9 TB Raw SSD (device paths: /dev/sdc) 10000 Mbps 
 RAM-intensive bare metal with SDS, mb4c.20x64.2x1.9tb.ssd: If you need extra local storage for performance, use this disk-heavy flavor that supports software-defined storage (SDS). Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 64 GB 2 TB HDD / 960 GB SSD 2 disks, 1.9 TB Raw SSD (device paths: /dev/sdc, /dev/sdd) 10000 Mbps 
 RAM-intensive bare metal with SDS, mb4c.32x384.3.8tb.ssd: If you need extra local storage for performance, use this disk-heavy flavor that supports software-defined storage (SDS). Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 32 / 384 GB 2 TB HDD / 960 GB SSD 3.8 TB Raw SSD (device paths: /dev/sdc) 10000 Mbps 
 RAM-intensive bare metal with SDS, mb4c.32x384.6x3.8tb.ssd: If you need extra local storage for performance, use this disk-heavy flavor that supports software-defined storage (SDS). Maximize the RAM available to your worker nodes.
However, that is where the similarities end.

Replication keeps your data in sync in two different locations. Only one of the volume pair (primary volume and replica volume) can be active at a time. The replication process automatically copies information from the active volume to the inactive volume based on the replication schedule. For more information about replica volumes, see [Replicating data](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-replication).

Duplication creates a copy of your volume based on a snapshot in the same availability zone as the parent volume. The duplicate volume inherits the capacity and performance options of the original volume by default and has a copy of the data up to the point-in-time of a snapshot. The duplicate volume can be dependent or independent from the original volume, and it can be manually refreshed with data from the parent volume. You can adjust the IOPS or increase the volume size of the duplicate without any effect on the parent volume.



* A dependent duplicate volume does not go through the conversion of becoming independent, and can be refreshed at any time after it is created. It locks the original snapshot so that snapshot cannot be deleted while the dependent duplicate exists. The parent volume cannot be canceled while the dependent duplicate volume exists. If you want to cancel the parent volume, you must either cancel the dependent duplicate first or convert it to an independent duplicate.
* An independent duplicate is superior to the dependent duplicate in most regards, but it cannot be refreshed immediately after creation because of the lengthy conversion process. It can take up to several hours based on the size of the volume. For example, it might take up to a day for a 12-TB volume. However, after the separation process is complete, the data can be manually refreshed by using another snapshot of the original parent volume.



For more information about duplicates, see [Creating and managing duplicate volumes](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-duplicatevolume).



Table 1. Comparison of features between different types of volume copies.
This table has row and column headers. The row headers identify the capability.