vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Benefits of key rotation 

Rotating your root keys provides these security benefits:



* You can limit how long your data is protected by a single root key. By shortening the time that your resources are encrypted with a particular root key, you decrease the probability of a security breach.
* You limit the number of resources encrypted with any one version of the root key because resources created after key rotation are encrypted with different key material.
* If your organization detects a security threat, you can immediately rotate your keys to protect your resources.
* Rotating keys mitigates costs that are associated with key compromises.







 Key rotation options 

You can rotate your root keys in your KMS instance by setting a rotation policy or by manually rotating your keys. A rotation policy automatically rotates your keys based on a schedule; manual rotation rotates keys on demand.

If you're using Hyper Protect Crypto Services, you can also rotate master keys by using [IBM Cloud TKE CLI plug-in](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-rotate-master-key-cli). Information in the following sections pertains to rotating root keys only.

You can set a [rotation policy](https://cloud.ibm.com/docs/key-protect?topic=key-protect-set-rotation-policy) for a KMS-generated root key and define a rotation schedule to automatically rotate your key. In your KMS instance, you can set a rotation interval in the range of 1 - 12 months.

You can also manually rotate your imported or KMS-generated root keys at any time. Key Protect and Hyper Protect Crypto Services allow one rotation per hour for each root key. The KMS replaces the key immediately upon your request. Use this option when you import a new root key to the KMS instance and want to rotate the key immediately.

For more information about these options, see [Comparing your key rotation options in Key Protect](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-rotationcompare-key-rotation-options).
* Vault: This option is used for workloads or cool data that are accessed infrequently, such as once a month or less. Common use cases are archives, short-term data retention, digital asset preservation, tape replacement, and disaster recovery.
* Cold: This option is used for cold data that is rarely accessed (every 90 days or less), or inactive data. Common use cases are archives, long-term backups, historical data that you keep for compliance, or workloads and apps that are rarely accessed.
* Flex: This option is used for workloads and data that don't follow a specific usage pattern, or that are too huge to determine or predict a usage pattern.



[Check out this blog](https://www.ibm.com/blogs/cloud-archive/interconnect-2017-changing-rules-storage) to learn how the Flex storage class works compared to traditional storage tiers.
3. Decide on the level of resiliency for the data that is stored in your bucket.



* Cross-region: With this option, your data is stored across three regions within a geolocation for highest availability. If you have workloads that are distributed across regions, requests are routed to the nearest regional endpoint. The API endpoint for the geolocation is automatically set by the ibmc Helm plug-in that you installed earlier based on the location that your cluster is in. For example, if your cluster is in US South, then your storage classes are configured to use the US GEO API endpoint for your buckets. For more information, see [Regions and endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpointsendpoints).
* Regional: With this option, your data is replicated across multiple zones within one region. If you have workloads that are located in the same region, you see lower latency and better performance than in a cross-regional setup. The regional endpoint is automatically set by the ibm Helm plug-in that you installed earlier based on the location that your cluster is in.
By default, all information that is stored in Schematics is encrypted in transit and at rest. To ensure resiliency and high availability, all data stored in the US and EU geographies is replicated across multiple locations in the same geography. When choosing a Schematics location to work with, verify that your data can be stored in these geographic locations.



Location information

 Geography/ location API endpoint Data stored Data replicated 

 North America Public <br>https://us.schematics.cloud.ibm.com <br> <br>https://cloud.ibm.com/schematics/overview <br> <br>Private <br>https://private-us.schematics.cloud.ibm.com (Deprecated) Workspaces that are created with this endpoint and all associated data are stored in the US. Data is replicated between two locations in the US. 
 Dallas Public <br>https://us-south.schematics.cloud.ibm.com <br> <br>Private <br>https://private-us-south.schematics.cloud.ibm.com Workspaces that are created with this endpoint and all associated data are stored in the Dallas location. Data is replicated between two locations in the US. 
 Washington Public <br>https://us-east.schematics.cloud.ibm.com <br> <br>Private <br>https://private-us-east.schematics.cloud.ibm.com Workspaces that are created with this endpoint and all associated data are stored in the Washington location. Data is replicated between two locations in the US. 
 Europe Public <br>https://eu.schematics.cloud.ibm.com <br> <br>Private <br>https://private-eu.schematics.cloud.ibm.com (Deprecated) Workspaces that are created with this endpoint and all associated data are stored in Europe. Data is replicated between two locations in Europe.
* \Memory\Available MBytes
* \SQLServer:Memory Manager\Target Server Memory (KB).
* \SQLServer:Memory Manager\Total Server Memory (KB).
* \SQLServer:Buffer Manager\Lazy writes/sec.
* \SQLServer:Buffer Manager\Page life expectancy.







 Storage 

The performance of a SQL Server instance is heavily dependent on storage performance, which is measured by IOPS and throughput counters. If your database does not completely fit into the available memory space, SQL Server moves database pages in and out of the buffer pool. Access to data files and log files is characterized as follows:



* Access to log files is sequential, except when a transaction must be rolled back
* Data files, including tempdb, are randomly accessed.



The following Windows Performance Monitor counters can help analyze the IO throughput required by your SQL Server:



* \LogicalDisk\Disk Reads/Sec - Read IOPS.
* \LogicalDisk\Disk Writes/Sec - Write IOPS.
* \LogicalDisk\Disk Read Bytes/Sec - Read throughput requirements for the data, log, and tempdb files.
* \LogicalDisk\Disk Write Bytes/Sec - Write throughput requirements for the data, log, and tempdb files.



I/O unit sizes influence both IOPS and throughput capabilities as:



* Smaller I/O sizes yield higher IOPS.
* Larger I/O sizes yield higher throughput.



SQL Server chooses the optimal I/O size automatically. Refer to [How block size affects performance](https://cloud.ibm.com/docs/vpc?topic=vpc-capacity-performancehow-block-size-affects-performance). Additionally, refer to [Storage-compute performance metrics](https://cloud.ibm.com/docs/vpc?topic=vpc-capacity-performancestorage-performance-metrics), which describes baseline metrics you can expect for read and write operations between your compute instances and block storage volumes.
The ubiquitous support for such canonicalization mitigates the downsides, and together with a developer's expectation that a full industry-standard format would be supported, the benefits exceed the risks in such a case.







 Sanitation and validation 

Definitively invalid or ambiguous input MUST NOT be ignored and therefore MUST result in an error. Validation is better than sanitation.

There are several reasons ignoring invalid input is dangerous and ill-advised. The following are hazards of a service that ignores invalid input:



* Client developers could be confused as to why a feature is not working as expected, when in reality their request is inadvertently malformed in a way that's hard to spot.
* Subtle bugs in client applications could result in dangerous unintentional consequences. For example, if a filter parameter on a list operation is ignored because a malformed value is unintentionally provided, a client might misconfigure or delete the wrong set of resources. In extreme cases such bugs could result in outages of dependent services, loss of data, or accidental access grants to resources.
* Subtle bugs in service code that fail to fully discard invalid input in all contexts could result in attack vectors.







 Extraneous input 

Except where required by an industry standard or convention

[3], extraneous input, such as unrecognized properties in JSON request bodies and unrecognized query parameters, SHOULD NOT be ignored and SHOULD result in an error.

All of the downsides listed for [silently ignoring invalid input](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-robustnesssanitation-and-validation) also apply to ignoring extraneous input.

Additionally, ignoring extraneous input causes particular hazards for evolving a service in a backward-compatible way. Specifically, if a client is sending a property or parameter that is unsupported and ignored, then the client application could experience a failure or an outage if and when support is added to the service for a property or parameter of the same name.







 Migrating to best practices for robustness
Users who have READ access to the base location of an index can use it by setting their base location. However, it is important to avoid multiple users who write indexes for the same data set to the same base location. Users can avoid sharing indexes by using different base locations.

Data Engine charges for index creation based on the amount of data scanned. Index creation for Parquet files benefits from the availability of schema information and the possibility of column projection to limit the amount of data scanned. However, index creation for JSON and CSV files requires schema inference that results in extra passes on the input data. To avoid extra passes on the data, create a table in the catalog for the respective data to provide the required schema information.





 Describing an index 

To retrieve data skipping index statistics and metadata, use the DESCRIBE operation, as in the following example:

DESCRIBE METAINDEX
ON cos://us-geo/sql/metergen STORED AS parquet

The result includes how many objects were indexed, whether the index is up to date, the base location of the indexes, and the index types that were generated.





 Using data skipping indexes 

When you create a data skipping index, Data Engine automatically uses it when it runs queries. The UI shows the percentage of skipped objects. You also find examples in the UI of queries that benefit from data skipping.





 Geospatial data skipping 

Data skipping is supported for queries that use [geospatial functions](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_functions.html).

The following list shows supported geospatial functions:



* ST_Distance
* ST_Intersects
* ST_Contains
* ST_Equals
* ST_Crosses
* ST_Touches
* ST_Within
* ST_Overlaps
* ST_EnvelopesIntersect
* ST_IntersectsInterior





 Geospatial data skipping with MinMax indexes 

You can use MinMax indexes on latitude and longitude columns for data skipping.
[Securing data using context-based restrictions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-cos-tutorial-cbr)Securing data using context-based restrictions

In this tutorial, you will establish context-based restrictions that prevent any access to object storage data unless the request originates from a trusted network zone.

Object Storage



* 10 minutes
* 2022-05-02



[Tracking Object Storage events in Activity Tracker](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-tracking-cos-events)Tracking Object Storage events in Activity Tracker

Tracking IBM Cloud® Object Storage events with IBM Cloud® Activity Tracker provides a record of what is happening with your data.

Object Storage



* 30 minutes
* 2022-06-24



[Monitoring Object Storage activity in Monitoring](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-monitoring-cos)Monitoring Object Storage activity in Monitoring

Tracking IBM Cloud® Object Storage events with IBM Cloud® Activity Tracker provides insights and information about what is happening with your data.

Object Storage



* 30 minutes
* 2020-04-27



[Create a Secure Content Store](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-secure-content-store)Create a Secure Content Store

Are you looking to store content securely (locally or globally) at an affordable cost, for things like cloud native apps, media storage, backup storage and archive data? IBM Secure Content Store powered by IBM Cloud® Object Storage provides unparalleled agility in supporting fast, highly consistent application deployment around the world to help customers securely expand their business into new regions, from business-critical data to video archive solutions. It also offers immutable storage, immutable backup, and archive data with industry-leading security and controls for regulatory/compliance requirements.

Object Storage



* 15 minutes
* 2023-06-29
Schematics supports both Terraform and Ansible as IaC tools with Schematics Workspaces and Actions. When lifecycle management is important with environments being regularly stood up and torn down, using Terraform with Schematics Workspaces is recommended. Terraform keeps a record of the current state of your deployed cloud infrastructure and Schematics is able to remove your infrastructure in reverse dependency order without manual intervention.





 Idempotence 

A benefit of the declarative approached used by Terraform and Ansible is idempotence. Idempotent tasks can be executed multiple times with the same end result. Irrespective of the previous state or starting place when restarting after failures, the provisioned infrastructure and configuration are always the same. This aspect is key to ensuring consistency and repeatability of environments deployed using Schematics.

How you use a tool and the modules used both have an impact on idempotency. Generally, Terraform and Ansible modules are written to be idempotent. With both tools, code can we written that does not yield an idempotent result. In which case, the configuration may drift from the desired target state. With Terraform this form of drift is most likely when null-resources are used to extend provider functionality with custom scripts which are not idempotent.

Immutability is an IaC practice that minimizes the risk of drift from the target state.





 Immutablity 

Immutable infrastructure refers to managing services and software deployments where resources like containers or virtual machines are replaced rather than changed (using scripts). The main desire here for immutability is avoiding configuration drift. Inconsistencies that arise due to local or manual changes, or differences in the sequence of automated operations. Changes that make it harder to debug and resolve issues, and increase support costs.

To ensure immutability and eliminate drift, all changes should be made through the Schematics IaC configuration, and resources like VSIs should be redeployed when they need updating.







 Next steps 

Now you understand more about IaC, why not review the use of IaC in Schematics:
* UI
* CLI







 Replicating Data 

Replication uses one of your snapshot schedules to automatically copy snapshots to a destination volume in a remote data center. The copies can be recovered in the remote site if a catastrophic event occurs or your data becomes corrupted.

Replication keeps your data in sync in two different locations. If you want to clone your volume and use it independently from the original volume, see [Creating a duplicate File Volume](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-duplicatevolume).

Before you can replicate, you must create a snapshot schedule. The option to Order Replica does not appear until this condition is met.



 Determining the remote data center for the replicated storage volume in the UI 

IBM Cloud® data centers are paired into primary and remote combinations in every region worldwide. When you replicate data, consider the local data residency laws because moving data across borders can have legal implications. Replication across regions is not permitted.

See Table 1 for the complete list of data center availability and replication targets within each region.



Table 1 - This table shows the complete list of data centers with enhanced capabilities in each region. Every region is a separate column. Some cities, such as Dallas, San Jose, Washington DC, Amsterdam, Frankfurt, London, and Sydney have multiple data centers.

 US 1 US 2 Latin America Canada Europe Asia-Pacific Australia
* Delimiter
* Header
* File encoding
* Line delimiter
* Escape characters



* Ingestion of a single, multiple file(s), or a single folder (no sub folders) of S3 and local Parquet file(s).
* Ingestion of a single, multiple file(s), or a single folder (no sub folders) of S3 and local CSV file(s).



Following are some of the requirements or limitations of the ibm-lh tool:



* Source files must have the same type of format.
* Schema evolution is not supported.
* Iceberg target table is the supported output format.
* Partitioning is not supported.
* IBM Cloud Object Storage (COS), AWS S3, and MinIO object storage are supported.
* pathStyleAccess property for object storage is not supported.
For more information about each storage class, see the [storage class reference](https://cloud.ibm.com/docs/containers?topic=containers-block_storageblock_storageclass_reference). If you don't find what you are looking for, consider creating your own customized storage class. To get started, check out the [customized storage class samples](https://cloud.ibm.com/docs/containers?topic=containers-block_storageblock_custom_storageclass).
3. Choose the type of block storage that you want to provision.



* Bronze, silver, and gold storage classes: These storage classes provision Endurance storage. With Endurance storage, you can choose the size of the storage in gigabytes at predefined IOPS tiers.
* Custom storage class: This storage class provisions Performance storage. With performance storage, you have more control over the size of the storage and the IOPS.



4. Choose the size and IOPS for your block storage. The size and the number of IOPS define the total number of IOPS (input/ output operations per second) that serves as an indicator for how fast your storage is. The more total IOPS your storage has, the faster it processes read and write operations.



* Bronze, silver, and gold storage classes: These storage classes come with a fixed number of IOPS per gigabyte and are provisioned on SSD hard disks. The total number of IOPS depends on the size of the storage that you choose. You can select any whole number of gigabyte within the allowed size range, such as 20 Gi, 256 Gi, or 11854 Gi. To determine the total number of IOPS, you must multiply the IOPS with the selected size. For example, if you select a 1000Gi block storage size in the silver storage class that comes with 4 IOPS per GB, your storage has a total of 4000 IOPS.



Table of storage class size ranges and IOPS per gigabyte

 Storage class IOPS per gigabyte Size range in gigabytes 

 Bronze 2 IOPS/GB 20-12000 Gi 
 Silver 4 IOPS/GB 20-12000 Gi 
 Gold 10 IOPS/GB 20-4000 Gi
Now consider whether they would even notice when it takes a few seconds for a database update to propagate throughout the application.

Availability trumps consistency more than you might expect. Online shopping cart systems, HTTP caches, and DNS are a few more examples. Organizations must consider the cost of downtime such as user frustration, productivity loss, and missed opportunities.





 From theory to implementation 

Addressing high availability is vital for cloud applications. Otherwise, global database consistency stays a major bottleneck as you scale. Highly available applications need to maintain constant contact with their data, even if that data isn't up to date. That's the concept of eventual consistency, and it's nothing to be scared of. Sometimes, with a large scale, it's better to serve answers that aren't perfectly correct than to not serve them at all.

Database systems hide the complexities of availability versus consistency in different ways, but they always exist. IBM Cloudant, CouchDB, and other NoSQL database teams believe that the best policy requires developers to address these complexities early in the design process. By doing the hard work up front, you reduce surprises because applications are ready to scale from day one.
: Changes made to object storage data can be used as an [event source for Cloud Functions](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-functions).





 28 August 2019 

New feature!
: Data can be encrypted [using HPCS](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption).





 7 August 2019 

Activity tracking update
: Object-level events can be tracked [using Activity Tracker](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-at).





 11 June 2019 

New feature!
: Objects can be automatically deleted by [adding expiration rules to a bucket's lifecycle configuration](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry).





 15 May 2019 

New CLI plug-in!
: Users can access and interact with object storage [using the IBM Cloud CLI](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-cli-plugin-ic-cos-cli).





 26 April 2019 

New location!
: Buckets can now be created in a Single Data Center configuration in Hong Kong. More information can be found in the [Select Regions and Endpoints](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints).





 28 March 2019 

New feature!
: User can use [COS Firewall](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-setting-a-firewallsetting-a-firewall) to restrict access to the data in COS only if request originates from a list of allowed IP addresses.

IAM update