vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Best practices for File Storage for Classic 

Follow our best practices to maximize the performance of your storage, and avoid application downtime.



 Best practice 1 - Clear the path 

To achieve maximum IOPS, adequate network resources need to be in place.



* Run storage traffic on a dedicate VLAN. Running storage traffic through software firewalls increases latency and adversely affects storage performance. It's best to run storage traffic on a VLAN, which bypasses the firewall. For more information, see [routing storage traffic to its own VLAN interface](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-file-storage-faqshowtoisolatedstorage).
* Avoid routing your storage traffic to a gateway device whenever possible. When storage traffic is routed to a gateway device, it can add latency to storage traffic, or it can cause storage traffic disruption if the firewall in the gateway device is misconfigured. The storage disruption is especially true when maintenance such as a restart is required on a single (not clustered) gateway device. If a storage traffic must be routed through a gateway device, ensure that the gateway device has an at-least 10-Gbps interface, or the gateway device might become a network bottleneck.
* Use the subnet address instead of individual IP addresses on firewalls for NFS traffic. NFS hostname serves 6 IP addresses, and these 6 IP addresses can change.
* Use a faster NIC. Some limits are set at the LUN level and a faster interface doesn't increase that limit. However, with a slower Ethernet connection, your bandwidth can be a potential hindrance to achieving best performance levels.
* Choose a higher bandwidth. The speed of your Ethernet connection must be faster than the expected maximum throughput from your volume. Generally, don't expect to saturate your Ethernet connection beyond 70% of the available bandwidth.

For example, if you have 6,000 IOPS and are using a 16-KB block size, the volume can handle approximately 94-MBps throughput. However, when you have a 1-Gbps Ethernet connection to your LUN, it becomes a bottleneck when your servers attempt to use the maximum available throughput.
Effectively limits the amount of time data can remain unarchived.



[log_min_duration_statement](https://www.postgresql.org/docs/current/runtime-config-logging.html)



* Default - 100
* Restarts database - No
* Options - Minimum value of 100
* Notes - Statements that take longer than the specified number of milliseconds are logged.



The next three settings wal_level, max_replication_slots, and max_wal_senders enable use of the [wal2json logical decoding plug-in](https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-wal2json). Anyone not using this plug-in should leave these settings at the default.

[wal_level](https://www.postgresql.org/docs/current/runtime-config-wal.html)



* Default - hot_standby
* Restarts database - YES
* Notes - Controls WAL level. Allowed values are hot_standby or logical. Set to logical to use logical decoding. If you are not using logical decoding, using logical increases the WAL size, which has several disadvantages and no real advantage. If you check your configuration by using SHOW wal_level; in psql, note: as of version 9.6 the value hot_standby is mapped to replica.



[max_replication_slots](https://www.postgresql.org/docs/current/runtime-config-replication.html)



* Default - 10
* Restarts database - YES
* Notes - The maximum number of simultaneously defined replication slots. The default and minimum number of slots is 10. Twenty slots are reserved for internal use by your deployment for High-Availability (HA) purposes. To use slots, you need to set the value above 20 and have one slot per consumer. It is recommended to add one extra slot over the minimum per expected consumer.
For more information, see [Availability and Durability of File Storage for Classic](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-storageavailability).
* What's the average uptime for File Storage for Classic?

When you store your data in File Storage for Classic, it's durable, highly available, and encrypted. File Storage is built upon best-in-class, proven, enterprise-grade hardware and software to ensure high availability and uptime. To ensure that the availability target of 99.999% (five 9's) is met, the data is stored redundantly across multiple physical disks on HA paired nodes. Each storage node has multiple paths to its own Solid-State Drives and its partner node's SSDs as well. This setup protects against path failure, and also controller failure because the node can still access its partner's disks seamlessly. For more information, see [Availability and Durability of File Storage for Classic](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-storageavailability).
* Can I get storage performance metrics (IOPS or latency) from the Support teams?

IBM Cloud® does not provide storage performance IOPS and latency metrics. Customers are expected to monitor their own File Storage for Classic devices by using their choice of third-party monitoring tools.

The following examples are utilities that you might consider to use to check performance statistics.



* [sysstat](https://github.com/sysstat/sysstat/blob/master/README.md) - System performance tools for the Linux® operating system.
* [typeperf](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/typeperf) - Windows® command that writes performance data to the command window or to a log file.
* UI
* CLI







 Replicating Data 

Replication uses one of your snapshot schedules to automatically copy snapshots to a destination volume in a remote data center. The copies can be recovered in the remote site if a catastrophic event occurs or your data becomes corrupted.

Replication keeps your data in sync in two different locations. If you want to clone your volume and use it independently from the original volume, see [Creating a duplicate Block Volume](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-duplicatevolume).

Before you can replicate, you must create a snapshot schedule. The option to Order Replica does not appear unless this condition is met.



 Determining the remote data center for my replicated storage volume in the UI 

IBM Cloud® data centers are paired into primary and remote combinations in every region worldwide. When you replicate data, consider the local data residency laws because moving data across borders can have legal implications. Replication across regions is not allowed.

See Table 1 for the complete list of data center availability and replication targets within each region.



Table 1 - This table shows the complete list of data centers with enhanced capabilities in each region. Every region is a separate column. Some cities, such as Dallas, San Jose, Washington DC, Amsterdam, Frankfurt, London, and Sydney have multiple data centers.

 US 1 US 2 Latin America Canada Europe Asia-Pacific Australia
* path: Path from which the cached content is served. The default path is /* When used by the updateOriginPath API, this attribute refers to the new path to be added.
* performanceConfiguration: Specifications for the performance configuration of the mapping. We support the following performance configurations:



* General web delivery: The default performance option to optimize the general web page contents, such as the HTML, pictures, CSS, JS files.
* Dynamic content acceleration: Optimizes the forward path to the origin server using Akamai’s “SureRoute” feature, and supports embedded object pre-fetching and situational image compression.
* Video on demand optimization: Optimizes cache/offload and media retrieval path through the Akamai network, MIME types, network timeout conditions.
* Large file optimization: Optimizes for delivery of large file downloads with improved cache/offload (partial object caching with pre-fetched object data), object retrieval path.



* cacheKeyQueryRule: The following options are available to configure Cache Key behavior:



* include-all: Include all query arguments
* ignore-all: Ignore all query arguments
* ignore: space separated query-args: Ignores those specific query arguments. For example, ignore: query1 query2
* include: space separated query-args: Includes those specific query arguments. For example, include: query1 query2



* geoblockingRule



The following attributes are specific to the Mapping class:



* uniqueId: A 10-digit, system-generated, identifier that is unique to each mapping. Generated when a mapping is created.
* domain: Primary CDN name. Also referred to as hostname.
* protocol: Protocol used to set up services. Can be HTTP, HTTPS, or a combination of the two, HTTP_AND_HTTPS.
* cname: Canonical name record aliases the hostname. Can be provided by the user, or system-generated. If user-provided, it should be less than 64 alphanumeric characters and must be unique. If not provided, one is generated when the mapping is created.
About Object Storage for Satellite 

IBM Cloud Object Storage for IBM Cloud Satellite offers users the flexibility to run a managed Object Storage service on client-owned on-premises infrastructure, edge locations or third-party public cloud infrastructure.

This introductory offering of Object Storage for Satellite is limited in capabilities and will be expanded on in the future. Keep in mind that not all APIs or connected services may work in the same fashion as Object Storage on IBM Cloud.

Essentially, provisioning an instance of Object Storage for Satellite provides the same familiar interfaces of IBM Cloud Object Storage outside of IBM Cloud.

Object Storage is integrated into Satellite in three different ways: configuration data and backup storage for the Satellite instance itself, as a persistent volume that allows for file-like access, and as a local instance of an IBM Cloud Object Storage service instance. This documentation focuses on the latter - setting up and accessing an instance of Object Storage running on Satellite hardware.



 Typical use cases of Object Storage for Satellite 

Low latency workloads that need to be run in close proximity to on-premises data and applications including workloads running on factory floors for automated operations in manufacturing, real-time patient diagnosis, and media streaming.

Data residency requirements or those in regulated industries that need to securely store and process customer data that needs to remain on-premises or in locations where there is no public Cloud Object Storage service.

Edge or IOT applications that collect and process data on the edge of network for new workloads from devices and users such as data collection and processing, location-based media, autonomous vehicle data, analytics, and machine data controls for manufacturing.

Hybrid workloads that require management of data between on-premises infrastructure, edge, public cloud or any multi-cloud installation.





 How Object Storage for Satellite works 

Zoom

![COS on Satellite Architecture](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/satellite-arch.png)

Figure 1. Object Storage for Satellite Architecture



1. A Satellite administrator needs to configure a new "Location" using the Satellite console and assigns hosts for the Satellite Control Plane.
2.
[Memory](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-about-virtual-server-profilesmemory) Best for memory caching and real-time analytics workloads. 





 Limitations 

Consider the following limitations before you reserve capacity and provision a reserved virtual server instance:



* You can't use variable compute profiles.
* You can't upgrade or downgrade your instances.
* Reserved capacity cannot be canceled; however, you can reclaim virtual server instances in that capacity.







 Notifications 

You receive an email notification one month before the end of the term on your reserved virtual server capacity.





 Next steps 

After you review and decided on your options, it's time to provision your reserved capacity and instances. To get started, review the following information:



1. [Provisioning reserved capacity and instances](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-provisioning-reserved-capacity-and-instancesprovisioning-reserved-capacity-and-instances)
2. [FAQs: Reserved capacity and instances](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-faqs-reserved-capacity-and-instancesfaqs-reserved-capacity-and-instances)
MapReduce views are built asynchronously and can take some time to be ready for large data sets.

See some tips:

Use logic in your JavaScript to include only data that makes sense, for example, totalize only completed orders. Indexed keys don't have to be strings. A common pattern is to use array keys, for example, an array of year, month, day. These index keys allow query-time grouping by elements in the array. For example, you can group orders by year, orders by year and month, orders by year and month and day. Great for summary reports that allow the user to drill down into the detail. The value can by a string, number, or sometimes a small object that contains a subset of the document. The object can be used instead of adding include_docs=true, which would also return the document's body in the result set.

To summarize, MapReduce is a low-level means of defining indexes that allow the selection and aggregation of data.

Use JavaScript logic to decide which data makes it to the index. Choose how the index is formed by emitting keys-values.

Summarize data with the built-in reducers. Produce concise reports from lots of data efficiently.

MapReduce is great for boilerplate queries that your application needs to do again and again. Not for one-time, adhoc-queries for data exploration.

That's the end of this part. The next part is called Dates.







 Dates video 

Learn about different options for storing a date, or date and time value.



* Dates video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 13 - Dates.

We saw earlier in this course that JSON only natively models strings, numbers, booleans, objects, and arrays. A common use-case is to store a date or date and time value in a database. See some ideas on how that can be achieved with IBM Cloudant.
You have the option of using your existing instance of IBM Cloud Object Storage or creating a new instance. If you want to reuse your existing instance, skip to step #2.



1. Create an instance of IBM Cloud Object Storage from the [catalog](https://cloud.ibm.com/catalog/services/cloud-object-storage).
2. Create any buckets that you need to store your transferred data. Read through the [getting started guide](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage) to familiarize yourself with key concepts such as [endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints) and [storage classes](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-classes).
3. The rclone utility will not copy any bucket configurations or object metadata. Therefore, if you are using any of the Object Storage features such as expiration, archive, key protect, etc. be sure to configure them appropriately before migrating your data. To view which features are supported at your COS destination, please refer to the [feature matrix](https://cloud.ibm.com/docs/services/cloud-object-storage?topic=cloud-object-storage-service-availability).



Feature configuration and access policies documentation can be viewed at the IBM Cloud portal pages listed below:



* [IBM Cloud Identity and Access Management - IAM](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam)
* [Activity Tracker](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-at)
* [Metrics Monitoring](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-mm-cos-integration)
The value depends on whether you checked the High performance with Intel Optane box.





 Enable vSAN deduplication and compression 

vSAN storage depends on the number of servers and your total disk capacity, and the use of deduplication and compression.

The amount of storage reduction from deduplication and compression depends on many factors, including the type of data stored and the number of duplicate blocks. Larger disk groups tend to provide a higher deduplication ratio. For more information, see [Using deduplication and compression](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vsan.doc/GUID-3D2D80CC-444E-454E-9B8B-25C3F620EFED.html).





 vSAN license 

Use the IBM-provided VMware license for the vSAN component, which is included with purchase.

Bring Your Own License (BYOL) is no longer supported except for migrations or upgrades of existing BYOL clusters. Select I will provide or Use existing license and enter your own license key only if you are performing an upgrade or migration of an existing BYOL cluster.

The Use existing license option is available only if you are using a BYOL vSAN license for your instance. When the option is enabled, you can select the existing license only if the instance has enough capacity for the additional hosts.

If your initial cluster was a vSAN cluster, any additional vSAN clusters use the same vSAN license and have the same configuration as the initial one. This behavior is also true if any cluster (initial or additional) in the instance has vSAN chosen to be deployed on it. The first time you're prompted for the vSAN license and the edition. The next time that you select vSAN for a new cluster, the license that was chosen initially is reused.







 NFS storage 

When you select NFS storage, you can add file-level shared storage for your instance where all shares use the same settings or you can specify different configuration settings for each file share. The number of file shares must be in the range 1-100.

Specify the following NFS options.
If a greater level of organization is required within a bucket, the use of prefixes is supported: {endpoint}/{bucket-name}/{object-prefix}/{object-name}. The object's key remains the combination {object-prefix}/{object-name}.
* What is the difference between 'Class A' and 'Class B' requests?

'Class A' requests are operations that involve modification or listing. This includes creating buckets, uploading or copying objects, creating or changing configurations, listing buckets, and listing the contents of buckets.'Class B' requests are those related to retrieving objects or their associated metadata/configurations from the system. There is no charge for deleting buckets or objects from the system.
* What is the best way to structure your data by using Object Storage so you can 'look' at it and find what you are looking for? Without a directory structure, having thousands of files at one level is difficult to view.

You can use metadata that is associated with each object to find the objects you are looking for. The biggest advantage of Object Storage is the metadata that is associated with each object. Each object can have up to 4 MB of metadata in Object Storage. When offloaded to a database, metadata provides excellent search capabilities. Many (key, value) pairs can be stored in 4 MB. You can also use Prefix searching to find what you are looking for. For example, if you use buckets to separate each customer data, you can use prefixes within buckets for organization. For example: /bucket1/folder/object where 'folder/' is the prefix.
* Can you confirm that Object Storage is ‘immediately consistent’, as opposed to ‘eventually consistent’?

Object Storage is ‘immediately consistent’ for data and ‘eventually consistent’ for usage accounting.
* Can Object Storage partition the data automatically using HDFS, so I can read the partitions in parallel, for example, with Spark?

Object Storage supports a ranged GET on the object, so an application can do a distributed striped-read-type operation. Doing the striping is managed by the application.
* Can I restore a bucket from a specific back-up file?
Bare metal machines are optimized for different use cases such as data-, GPU-, or RAM-intensive workloads.

Choose a flavor, or machine type, with the correct storage configuration to support your workload. Some flavors have a mix of the following disks and storage configurations. For example, some flavors might have a SATA primary disk with a raw SSD secondary disk.



* SATA: A magnetic spinning disk storage device that is often used for the primary disk of the worker node that stores the OS file system.
* SSD: A solid-state drive storage device for high-performance data.
* SAN: For select virtual machines, the storage device is mounted via software area network (SAN).
* Raw: The storage device is unformatted and the full capacity is available for use.
* RAID: A storage device with data distributed for redundancy and performance that varies depending on the RAID level. As such, the disk capacity that is available for use varies.





Available bare metal flavors in IBM Cloud Kubernetes Service.

 Name and use case Cores / Memory Primary / Auxiliary disk Network speed 

 RAM-intensive bare metal, mb4c.20x64: Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 64 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 RAM-intensive bare metal, mb4c.20x192: Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT. 20 / 192 GB 2 TB HDD / 960 GB SSD 10000 Mbps 
 RAM-intensive bare metal, mb4c.20x384: Maximize the RAM available to your worker nodes. This bare metal includes 2nd Generation Intel® Xeon® Scalable Processors with Intel® C620 Series chip sets for better performance for workloads such as machine learning, AI, and IoT.
FROM cos://us-geo/sql/hospitals_geometry.parquet STORED AS PARQUET
WHERE ST_Intersects(ST_WKTToSQL(location), ST_Buffer(ST_WKTToSQL('POINT (-74.0 42.0)'), 46800.0))







 Choosing data formats 

You can use data skipping with all of the formats that are supported by Data Engine, except for AVRO. It is best for data layout to use a column-based format, such as Parquet. To infer the schema for CSV and JSON, before you run any queries, the entire data set must first be scanned. Scanning the entire data set is not necessary if you create tables by using the Data Engine [catalog](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-hivemetastore). Unlike Parquet and ORC, CSV and JSON do not have built-in data skipping capabilities and can potentially benefit more from data skipping.





 Refreshing data skipping indexes 

If data is added to a data set, or if modifications are made to a data set after the creation of a data skipping index, the new or changed data is not skipped during queries. When the amount of new data becomes significant, refresh the index incrementally, as follows:

REFRESH METAINDEX
ON cos://us-geo/sql/metergen STORED AS parquet





 Showing data skipping indexes 

To retrieve existing data skipping indexes under the currently configured metadata base location, use the SHOW operation, as in the following example:

SHOW METAINDEXES

The result includes the currently set metadata base location and a list of indexed data sets.

If you see DEPRECATED_SUPPORTED in front of the index, it means that the metadata version is deprecated but supported and skipping works. The next REFRESH updates the metadata automatically.





 Deleting data skipping indexes 

To delete a data skipping index, use the following query:

DROP METAINDEX
ON cos://us-geo/sql/metergen STORED AS parquet

Dropping a table does not also drop the table indexes.