vania-janet/multiturn-rag-retrieval-data huggingface dataset:
"retries_per_request": 20,
"http_connections": 30
}





 The effect of large attachments 

Having large numbers of attachments on documents might cause an adverse effect on replication performance.

For more information about the effect of attachments on replication performance, see [Performance considerations](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-attachmentsperformance-considerations).



 Avoiding the /_replicate endpoint 

Use the [_replicator scheduler](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationthe-replication-scheduler) instead of the /_replicate endpoint.

If a problem occurs during replication, such as a stall, timeout, or application crash, a replication that is defined within the _replicator database is automatically restarted by the system. However, if you define a replication by sending a request to the /_replicate endpoint, it can't be restarted by the system if a problem occurs because the replication request doesn't persist. Replications that are defined in the _replicator database are easier to monitor.
As the result, you're billed an overage charge based on 87 GB x 12 hours = 1044 GB hours for that extra space.

Overage is calculated by using the maximum number of GB more than the plan allocation during a particular hour within the billing cycle.







 Disk overage example 

Assume that you start a month of 30 days with a Standard plan service instance that uses 9 GB of storage. Next, your storage increases to 21.5 GB for 15 minutes during the hour beginning at 02:00 of day 3. The instance drops back to 9.5 GB for the next 10 minutes of hour 02:00, then increases to 108 GB for the next 25 minutes of hour 02:00. Finally, your instance finishes the hour and indeed the rest of the month by dropping down to 28 GB.

This pattern means the maximum number of GB more than the plan allocation was 88 GB during hour 2 of day 3. From hour 03:00 of day 3, and for the rest of the month, your instance was 8 GB more than the plan allocation.

Therefore, from hour 02:00 of day 3, your bill includes an overage based on 88 GB x 1 hour = 88 GB hours.

From hour 03:00 of day 3 to the end of day 3, your bill includes an overage based on 8 GB x 21 hours = 168 GB hours.

From hour 00:00 of day 4 until the end of the month (of 30 days), your bill includes an overage. The overage is based on 8 GB x 24 hours x 27 days = 5184 GB hours.

The total overage bill for the month is based on a total of 88 + 168 + 5184 = 5440 GB hours.







 Request and document size limits 

IBM Cloudant JSON documents and requests have the following maximum size limits:



Table 4. Maximum size limits for JSON documents and requests

 Limit Maximum Size 

 Individual Document Size 1 MB 
 Single Attachment Size 10 MB 
 Request Body Size 11 MB 



If you exceed these limits, a [413 response](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes) alerts you.
| Allocation per member 12288mb
| Minimum 36864mb
| Step Size 1024mb
| Adjustable true

+ CPU
| Allocation 18
| Allocation per member 6
| Minimum 18
| Step Size 1
| Adjustable true

+ Disk
| Allocation 61440mb
| Allocation per member 20480mb
| Minimum 20480mb
| Step Size 1024mb
| Adjustable true

The deployment has three members, with 36864 MB of RAM and 61440 MB of disk allocated in total. The "per member" allocation is 12288 MB of RAM and 20480 MB of disk. The minimum value is the lowest the total allocation that can be set. The step size is the smallest amount by which the total allocation can be adjusted.

The cdb deployment-groups-set command allows either the total RAM or total disk allocation to be set, in MB. For example, to scale the memory of the "example-deployment" to 13312 MB of RAM for each memory member (for a total memory of 39936 MB), you use the command

ibmcloud cdb deployment-groups-set example-deployment member --memory 39936





 Scaling in the API 

The Foundation Endpoint that is shown on the Overview pane of your service provides the base URL to access this deployment through the API. Use it with the /groups endpoint if you need to manage or automate scaling programmatically.

To view the current and scalable resources on a deployment, use

curl -X GET -H "Authorization: Bearer $APIKEY" 'https://api.{region}.databases.cloud.ibm.com/v4/ibm/deployments/{id}/groups'

To scale the memory of a deployment to 13312 MB of RAM for each memory member (for a total memory of 39936 MB).

curl -X PATCH 'https://api.{region}.databases.cloud.ibm.com/v4/ibm/deployments/{id}/groups/member' -H "Authorization: Bearer $APIKEY" -H "Content-Type: application/json" -d '{"memory": {
* ibmcloud is instance-volume-attachment-add data-vol-name 72251a2e-d6c5-42b4-97b0-b5f8e8d1f479 --profile general-purpose --source-snapshot eaf9d6ca-35bf-4ac7-bc45-d0f2507f2830 --auto-delete true --output JSON
* ibmcloud is instance-volume-attachment-add data-vol-name 72251a2e-d6c5-42b4-97b0-b5f8e8d1f479 --profile general-purpose ---capacity 10 --auto-delete true --tags my-tag-1,my-tag-2
* ibmcloud is instance-volume-attachment-add data-vol-name my-instance my-volume --auto-delete true Add an existing volume to a virtual server instance by using resource name.







 Command options 



* NAME: Name of the volume attachment.
* INSTANCE: ID or name of the instance.
* VOLUME: ID or name of the volume.
* --new-volume-name: The name of new volume.
* --profile: Name of the profile.
* --iops: Input/output operations per second for the volume, it is only applicable for custom profile volumes. For the available IOPS ranges, see [Custom IOPS profile] [Onboarding software to your account](https://cloud.ibm.com/docs/docs/vpc?topic=vpc-block-storage-profilescustom).
* --encryption-key: The CRN of the Key Management Service root key.
* --capacity: The capacity of the volume in gigabytes. Range 10 - 16000 for custom and general-purpose profile volumes, 10 - 9600 for 5iops-tier profile volumes, 10 - 4800 for 10iops-tier profile volumes.
* --tags: Comma-separated tags for the volume.
* --source-snapshot: ID or name of the snapshot to clone volume.
Getting started with File Storage for Classic 

IBM Cloud® File Storage for Classic is persistent, fast, and flexible network-attached, NFS-based File Storage for Classic. In this network-attached storage (NAS) environment, you have total control over your file shares function and performance. File Storage for Classic shares can be connected to up to 64 authorized devices over routed TCP/IP connections for resiliency.

For more information about using File Storage for Classic with the IBM Cloud® Kubernetes Service, see [Storing data on classic IBM Cloud File Storage](https://cloud.ibm.com/docs/containers?topic=containers-file_storage).



 Before you begin 

File Storage for Classic volumes can be provisioned from 20 GB to 12 TB with two options:



* Provision Endurance tiers that feature pre-defined performance levels and other features like snapshots and replication.
* Build a high-powered Performance environment with allocated input/output operations per second (IOPS).



For more information about the File Storage for Classic offering, see [What is IBM Cloud File Storage](https://www.ibm.com/products/file-storage).





 Step 1: Provisioning considerations 



 Block size 

The IOPS value for both Endurance and Performance is based on a 16-KB block size with a 50/50 read and write, 50/50 random and sequential workload. A 16-KB block is the equivalent of one write to the volume.

The block size that is used by your application directly impacts the storage performance. If the block size that is used by your application is smaller than 16 KB, the IOPS limit is realized before the throughput limit. Conversely, if the block size that is used by your application is larger than 16 KB, the throughput limit is realized before to the IOPS limit.



Table 1 shows examples of how block size and IOPS affect the throughput. Average IO size x IOPS = Throughput in MB/s.

 Block Size (KB) IOPS Throughput (MB/s) 

 4 1,000 4 
 8 1,000 8 
 16 1,000 16 
 32 500 16 
 64 250 16 
 128 128 16 
 512 32 16 
 1024 16 16 







 Authorized hosts
Data IO and encryption 

Object size can have significant impacts on IBM Cloud® Object Storage performance. Choose the right approach for your workload.



 Multipart transfers 

Under typical conditions, multipart uploads and downloads are a very efficient method for breaking up transfers into many parallel transactions. Depending on the object size, a part size of 100MB is generally recommended. In any case, it is most efficient to set the part size to a multiple of 4MiB in order to optimize the data ingest into and egress out of COS.

As with AWS S3, using multipart transfers provides the following advantages:



* Improved throughput — You can upload parts in parallel to improve throughput.
* Quick recovery from any network issues — Smaller part size minimizes the impact of restarting a failed upload due to a network error.
* Pause and resume object uploads — Upload object parts over time. Once a multipart upload is initiated a multipart upload there is no expiry; it must explicitly complete or the multipart upload has to be aborted.
* Begin an upload before the final object size is known — An object can be uploaded as it is being created.



Due to the additional complexity of multipart transfers, it is recommended to use appropriate S3 libraries, tools, or SDKs that offer support for managed multipart transfers:



* [IBM COS SDK for Java](https://github.com/IBM/ibm-cos-sdk-java)
* [IBM COS SDK for Python](https://github.com/IBM/ibm-cos-sdk-python)
* [IBM COS SDK for Javascript (Node.js)](https://github.com/IBM/ibm-cos-sdk-js)
* [IBM COS SDK for Go](https://github.com/IBM/ibm-cos-sdk-go)
* [IBM COS Plug-in for IBM Cloud CLI](https://github.com/IBM/ibmcloud-cos-cli)
ibm.io/quota-limit
: To use this annotation, you must specify the --set quotaLimit=true option during installation. If you want to use this annotation, but didn't specify --set quotaLimit=true during installation, [re-install the helm chart](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_cos_install).
: If ibm.io/quota-limit is set to true, your PVC will set a maximum amount of storage (in bytes) available for the bucket based on the size storage: <size> that you specify.
: If ibm.io/quota-limit is set to false, the quota is not enforced on your PVC meaning that actual amount of storage in bytes might exceed the storage: <size> that you specified depending on your app.

ibm.io/endpoint
: If you created your IBM Cloud Object Storage service instance in a location that is different from your cluster, enter the private or public cloud service endpoint of your IBM Cloud Object Storage service instance that you want to use. For more information and an overview of available service endpoints, see [Additional endpoint information](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-advanced-endpoints). By default, the ibmc Helm plug-in automatically retrieves your cluster location and creates the storage classes by using the IBM Cloud Object Storage private cloud service endpoint that matches your cluster location. If your classic cluster is in a multizone metro, such as dal10, the IBM Cloud Object Storage private cloud service endpoint for the multizone metro, for example Dallas. To verify that the service endpoint in your storage classes matches the service endpoint of your service instance, run oc describe storageclass < storageclassname>. Make sure that you enter your service endpoint in the format https://< s3fs_private_service_endpoint> for private cloud service endpoints, or http://< s3fs_public_service_endpoint> for public cloud service endpoints.
Encrypted storage that is provisioned in these data centers can be increased up to 12 TB.

Existing size limitations for File Storage for Classic that was provisioned with Endurance still apply (up to 4 TB for 10 IOPS tier and up to 12 TB for all other tiers).





 Resizing storage in the UI 



1. Go to the [IBM Cloud® console](https://cloud.ibm.com/login). From the menu, select Classic Infrastructure![Classic icon](https://cloud.ibm.com/docs-content/v1/content/04e9937a86546143babfc65ea21fdd4ea2d12d13/icons/classic.svg). Click Storage > File Storage for Classic.
2. Select the volume from the list and click the ellipsis ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/04e9937a86546143babfc65ea21fdd4ea2d12d13/icons/action-menu-icon.svg) > Modify File Share.
3. Enter the new storage size in GB.
4. Review your selection and the new pricing. Click Modify.
5. Your new storage allocation is available in a few minutes.



Unmount, then mount the modified volume, so the OS can recognize the extra storage space.





 Resizing storage from the SLCLI 

To increase your storage volume, you can use the following command in SLCLI.

 slcli file volume-modify --help
Usage: slcli file volume-modify [OPTIONS] VOLUME_ID

Options:
-c, --new-size INTEGER New Size of file volume in GB. If no size
is given, the original size of volume is
used.
Potential Sizes: [20, 40, 80, 100,
250, 500, 1000, 2000, 4000, 8000, 12000]
Minimum: [the original size of the volume]
-i, --new-iops INTEGER Performance Storage IOPS, between 100 and 6000
sc_30_nwd_1_tl_How_20HTTP_20works_20with_20IBM_20Cloudant_20_7C_20IBM_20Cloud_20Docs_p_https_3A_2F_2Fcloud.ibm.com_2Fdocs_2FCloudant_3Ftopic_3DCloudant-how-http-works-with-cloudant_r_lt_973_evt_pageLoad_sv_1_rn_398945](https://bat.bing.com/action/0?ti=146001191&Ver=2&mid=6476e991-b2bf-4d3f-8c2a-21794b99b515&sid=cca51480229211eeac887b94c5a6c542&vid=cca50460229211ee874f7dc32b9b9a8d&vids=0&msclkid=N&pi=918639831&lg=en-US&sw=3840&sh=2160&sc=30&nwd=1&tl=How%20HTTP%20works%20with%20IBM%20Cloudant%20%7C%20IBM%20Cloud%20Docs&p=https%3A%2F%2Fcloud.ibm.com%2Fdocs%2FCloudant%3Ftopic%3DCloudant-how-http-works-with-cloudant&r=&lt=973&evt=pageLoad&sv=1&rn=398945)
For 96,000 IOPS to be realized, a single file share must be accessed by multiple virtual server instances. A single file share in an instance is limited to 48,000 IOPS.







 Adjust IOPS in the UI 

Follow these steps to adjust IOPS by selecting a new IOPS tier or custom IOPS band:



1. Go to the list of file shares. In [IBM Cloud console](https://cloud.ibm.com/login), go to menu icon ![menu icon](https://cloud.ibm.com/docs-content/v1/content/icons/icon_hamburger.svg) > VPC Infrastructure > Storage > File shares. By default, file shares display for all resource groups in your region.
2. In the File shares for VPC list page, click the name of a file share to see its details.
3. On the file share details page, locate Profile and click the pencil icon or use the Actions menu and select Edit IOPS profile. Volumes must be attached to a virtual server instance for these actions.
4. In the side panel, adjust IOPS as follows:



* For an IOPS tier, select a different tier from the menu. For example, you might have a 3 IOPS/GB general-purpose profile that you want to increase to a 5 IOPS/GB profile.
* For a custom IOPS or dp2 profile, the current IOPS value is shown and file share size. Enter a new IOPS value within the allowable range for that size.



5. Review the estimated monthly order summary for your geography and new pricing.
6. If you're satisfied, click Save and continue.



Your new IOPS allocation is realized when you restart the instance.





 Adjust IOPS from the CLI 



 Adjust IOPS for a Custom or dp2 profile 

From the CLI, use the share-update command with the --iops property to indicate the new IOPS size for a custom or dp2 profile.
* ibmcloud is instance-volume-attachment-add data-vol-name 72251a2e-d6c5-42b4-97b0-b5f8e8d1f479 --profile general-purpose --source-snapshot eaf9d6ca-35bf-4ac7-bc45-d0f2507f2830 --auto-delete true --output JSON
* ibmcloud is instance-volume-attachment-add data-vol-name 72251a2e-d6c5-42b4-97b0-b5f8e8d1f479 --profile general-purpose ---capacity 10 --auto-delete true --tags my-tag-1,my-tag-2
* ibmcloud is instance-volume-attachment-add data-vol-name my-instance my-volume --auto-delete true Add an existing volume to a virtual server instance by using resource name.







 Command options 



* NAME: Name of the volume attachment.
* INSTANCE: ID or name of the instance.
* VOLUME: ID or name of the volume.
* --new-volume-name: The name of new volume.
* --profile: Name of the profile.
* --iops: Input/output operations per second for the volume, it is only applicable for custom profile volumes. For the available IOPS ranges, see [Custom IOPS profile] [Onboarding software to your account](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profilescustom).
* --encryption-key: The CRN of the Key Management Service root key.
* --capacity: The capacity of the volume in gigabytes. Range 10 - 16000 for custom and general-purpose profile volumes, 10 - 9600 for 5iops-tier profile volumes, 10 - 4800 for 10iops-tier profile volumes.
* --tags: Comma-separated tags for the volume.
* --source-snapshot: ID or name of the snapshot to clone volume.
However, in a real application, the page size might be different and depends on document size, latency demands, memory consumption, and other tradeoffs.You can use the options that are described in the following sections.<-- <section "id="section-option-1-fetch-one-doc-too-many" "> --> Option 1 - Fetch one document too many Instead of fetching five documents (limit=5), fetch 5+1 (limit=6), but hide the sixth document from your users. The _id of the sixth document becomes the startkey of your request for the next page of results.See the following example of a first request:<-- <ul> --> * Curl * Java * Node * Python * Go<-- </ul> --> curl -H "Authorization: Bearer $API_BEARER_TOKEN" "$SERVICE_URL/orders/_all_docs?limit=6"
import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.AllDocsResult;
import com.ibm.cloud.cloudant.v1.model.DocsResultRow;
import com.ibm.cloud.cloudant.v1.model.PostAllDocsOptions;

Cloudant service = Cloudant.newInstance();

int pageSize = 5;
PostAllDocsOptions.Builder docsOptionsBuilder =
new PostAllDocsOptions.Builder()
.db("orders")
.limit(pageSize + 1); // Fetch pageSize + 1 documents

AllDocsResult response =
service.postAllDocs(docsOptionsBuilder.build())
.execute()
.getResult();

while (response.getRows().size() > 1) {
List<DocsResultRow> responseDocuments = response.getRows();
// on the last page, show all documents:
if (responseDocuments.size() <= pageSize) {
System.out.println(responseDocuments);
ibmcloud is share-mount-target my-file-share-3 my-target1

Getting mount target ID my-target1 for share ID my-file-share-3 under account Test Account as user test.user@ibm.com...

ID r026-c33d31bd-50e0-4401-abba-9c46ca698cb9
Name my-target1
VPC ID Name
r026-b403cfe8-917e-4fb8-a72c-bb490c735119 vpc-1006-01

Lifecycle state stable
Mount path fsf-syd0151a-fz.adn.networklayer.com:/abb3339c_12b2_4e92_914c_02f1018b45d3
Created 2023-07-05T17:57:10+05:30







 View file shares and mount targets with the API 

As described in the [Beta VPC API](https://cloud.ibm.com/apidocs/vpc-beta) reference [versioning](https://cloud.ibm.com/apidocs/vpc-betaapi-versioning-beta) policy, support for older versions of the beta API is limited to 45 days. Therefore, beta API requests must specify a version query parameter date value within the last 45 days. You must also provide generation parameter and specify generation=2. For more information, see Generation in the [Virtual Private Cloud API reference](https://cloud.ibm.com/apidocs/vpcapi-generation-parameter).



 View all file shares with the API 

Make a GET /shares request to list all file shares for a region.

curl -X GET "$vpc_api_endpoint/v1/shares?version=2023-07-05?limit=50&generation=2&maturity=beta" -H "Authorization: $iam_token"

A successful response looks like the following example. In the example, the limit query parameter specifies a limit of 50 file shares, all though there is only one in the response.
This tier is available in [select data centers](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-selectDC) for storage that is provisioned up to 4 TB.
* The available custom IOPS range depends on the volume capacity. The following table shows the available IOPS ranges based on volume size.





Table 1. Available IOPS based on volume size

 Volume size (GB) IOPS range 

 10 - 39 100 - 1,000 
 40 - 79 100 - 2,000 
 80 - 99 100 - 4,000 
 100 - 499 100 - 6,000 
 500 - 999 100 - 10,000 
 1,000 - 1,999 100 - 20,000 
 2,000 - 2,999 200 - 40,000 
 3,000 - 3,999 200 - 48,000 
 4,000 - 7,999 300 - 48,000 
 8,000 - 9,999 500 - 48,000 
 10,000 - 12,000 1,000 - 48,000 



7. In the side panel, review your order summary, and apply your Promo Code if you have one.

Discounts are applied when the order is processed.
8. Review your order, and read the service agreement. If you agree with the terms, check the box.
9. Click Create. Your new storage allocation is available in a few minutes.



By default, you can provision a combined total of 700 File Storage for Classic volumes. To increase the number of your volumes, contact your sales representative. Read about increasing limits [here](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-managinglimits). For more information about the limit on simultaneous authorizations, see the [FAQs](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-file-storage-faqsauthlimit).





 Ordering File Storage for Classic from the SLCLI 

You can use the SLCLI to place orders for products that are normally ordered through the [IBM Cloud console](https://cloud.ibm.com/login).
Provisioned throughput capacity model FAQ 

IBM® Cloudant® for IBM Cloud® calculates your provisioned throughput capacity based on these operation types: Read, Write, and Global Query.



  How is provisioned throughput capacity calculated? 

IBM Cloudant calculates provisioned throughput capacity by totaling the usage for each request class per second, where 1 second is a sliding window. When an account exceeds the total number of a request class that is allotted by its plan, IBM Cloudant rejects subsequent requests of that request class. No new requests are accepted until the usage of that request class inside the sliding window falls under the allowed limit. The sliding 1-second window is any consecutive period of 1,000 milliseconds.

For example, the Standard plan instance limits you to 200 reads per second. When you exceed 200 read requests, IBM Cloudant rejects future read requests made during the sliding 1,000-millisecond window. Read requests resume when the number of read requests for that time period is less than 200.

Request class units do not necessarily have a one-to-one mapping with HTTP requests. A single HTTP request can consume multiple units of a request class or classes if, for example, it reads multiple documents or both reads and writes.





  What happens when I exceed the provisioned throughput capacity limit? 

When you exceed the number of allowed events, IBM Cloudant generates a [429 Too Many Requests](https://cloud.ibm.com/apidocs/cloudantlist-of-http-codes) response. You must make sure ahead of time that your applications can handle 429 responses.

If you use the most recent versions of the [client libraries](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-client-libraries) that IBM Cloudant supports, you can set up your applications to handle 429 responses. This step is important because most client libraries don't automatically attempt to retry a request when a 429 response occurs. You need to verify that your application handles 429 responses correctly because IBM Cloudant limits the number of retries. Regularly exceeding the number of requests indicates that you need to move to a different plan.