vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Previous profiles are not supported when you provision a new file share, which is based on the dp2 profile. However, earlier version file shares can continue to use existing profiles.
* Restricting file share access to specific virtual server instances and data encryption in transit is available only for shares that are based on the dp2 profile.
* Windows operating systems are not supported.
* Minimum capacity is 10 GB per file share.
* Maximum capacity is 32,000 GB per file share.
* No data retention policy exists for deleted file shares. You cannot undelete a file share after you delete it.
* Up to 256 hosts per zone per VPC can be concurrently connected to a single file share.







 Related information 

For more information about the IBM Cloud VPC, see [Virtual Private Cloud](https://www.ibm.com/cloud/learn/vpc).

For more information about VPC, see [Getting started with Virtual Private Cloud](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-started).





 Next steps 



* [Plan for creating file shares and mount targets](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-planning).
* [Create a file share and mount targets](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-create).
See [Understanding Contract Elements](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classification) for details.





 26 April 2018 

Fixed ingestion normalization known issue : The following ingestion issue is fixed: In some cases where post-enrichment json_normalizations and/or normalizations were specified, the normalizations might be applied in the wrong order, possibly resulting in documents being indexed with unexpected field values. This is no longer the case.

Change to maximum file size : The maximum file size for a sample document is now 1MB. The maximum file size was previously 5MB.





 12 April 2018 

New Knowledge Graph filters : Knowledge Graph: Evidence and Canonicalization and filtering are now available in all collections. In any collections created before 03-05-2018, you need to reingest your documents to use these features. Previously, you needed to create a new collection and reingest your documents. : Update: As of 30 September 2019, the IBM Watson™ Discovery Knowledge Graph Beta APIs are no longer accessible. See the [3 September 2019 release notes](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notesdiscovery-3september2019) for more information.





 11 April 2018 

New Element Classification categories : Two additional categories are now supported in Element Classification: Asset Use and Communication. See [Understanding Contract Elements](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classification) for details.





 2 April 2018 

Change to sample document deletion : Sample documents are now automatically deleted after 24 hours, instead of 1 month.





 16 March 2018 

New full support for German language : Added full support for German. For more information, see [Language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support).

New Element Classification configuration : A new Discovery tooling configuration named Default Contract Configuration is added to support Element Classification, which can be used to extract party, nature, and category from elements in PDFs.
<br>File Share Capacity can be expanded to 12 TB after initial provisioning in GB increments. 
 Protocols NFS 
 Host connections Maximum of 64 for NFS. 
 Use cases Tier 1, 2, and 3 workloads: <br>0.25 IOPS per GB: Low I/O intensity. Example applications include storing mailboxes or department-level file shares. <br>2 IOPS per GB: General purposes. Example applications include hosting small databases that are backing web applications or virtual machine disk images for a hypervisor. <br>4 IOPS per GB: High I/O intensity. Example applications include transactional and other performance-sensitive databases. <br>10 IOPS per GB: High I/O intensity. Example applications include analytics. 
 HA Yes, dual heads and controllers. 
 Configurability Size and IOPS only. 
 Disaster recovery and replication Snapshot and Replication provided over the IBM Cloud® Private Network, also achieved by using [vRealize Automation](https://www.vmware.com/products/vrealize-automation), but no SRA. 
 Reliability Highly redundant, MPIO connection, NFS-based file storage routed TCP/IP connections. Snapshots and Replication enabled. 
 Latency <10 ms 
 Cost High for both CapEx (10x for SAN of the same size) and OpEx 



For more information, see the following links.



* [File storage](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-getting-started)
* [Architecture Guide for IBM File Storage for IBM Cloud with VMware](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-architectureguide)
If you choose to use this file storage from IBM Cloud VPC, you can attach an NFS share to your selected ESXi hosts and VMware clusters. This share can be used for management components such as vCenter Server, VMware NSX-T managers and NSX-T edge nodes, or for other VMware workloads and VMs.

These file shares are accessible only within the zone in which you created them, for example us-south-1. They are identified by name and associated with a resource group in your IBM Cloud customer account.

You have two options for creating file shares, by using an IOPS tier profile or by using a custom IOPS profile.

IOPS tiers provide a guaranteed level of performance for your workloads. You can select from three tiers:



* 3 IOPS/GB
* 5 IOPS/GB
* 10 IOPS/GB



With custom IOPS profiles, you can choose the performance that you want, by selecting custom allocated I/O operations per second (IOPS). You can increase the file share size from its original capacity in GB increments up to 32,000 GB capacity, depending on your file share profile.

The storage is attached by using the NFS v4.1 protocol to your hosts. To create an NFS mount path, you need to create mount targets. A mount target for a file share is a network endpoint or path. When you create a mount target, an NFS mount path is created for the file share. You can create a mount target by providing a VPC or subnet information. If you want to connect a file share to instances that are running in multiple VPCs in the same zone, you can create multiple mount targets for different VPCs.

For more information about the shared storage, see [File Storage for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-vpc-about&interface=ui).







 Related links 



* [IBM Cloud VPC getting started](https://cloud.ibm.com/docs/vpc?topic=vpc-getting-started)
(https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_disk_iops_read_write_total) 
 [Max allowed memory for an instance.](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_memory_limit_bytes) 
 [Read replica replication lag.](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_read_replica_replication_lag_bytes) 
 [Successful archive rate](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_successful_archive_rate) 
 [Temporary files size in bytes](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_temp_bytes_count) 
 [Total disk space for an instance.](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_disk_total_bytes) 
 [Transaction commit count](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_transaction_commit_count) 
 [Transaction commit rate](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-monitoringibm_databases_for_mysql_transaction_commit_rate)
* If public virtual servers use balanced local storage, you can't detach primary or secondary disks.
* You can attach the disks to another server if the change doesn't exceed the disk quota or the maximum volume size limit of the targeted virtual server.
* Moved disks are converted to the storage type of the target server.
* When a portable storage volume is attached to a virtual server in a different data center than the original virtual server, IBM Cloud copies the volume to the SAN in the new data center. The system then verifies the integrity of the copied volume and removes the original portable volume from the original data center SAN.









 LVM limitations 

Logical volume management (LVM) isn't supported as a bootable partitioning scheme. With proper operating system support and configuration, you can use secondary virtual server disks for LVM partitions. However, LVM isn't a supported file system for image templates or flex images.





 Supplemental storage 

Virtual servers are fully compatible with File Storage for Classic and Block Storage for Classic, and, IBM Cloud® Object Storage. These storage types are recommended for cluster drives, shared file storage, archival storage, large storage requirements, or specific performance requirements.

For more information about supplemental storage options, see the following resources:



* [Getting Started with Block Storage](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-started)
* [Getting Started with File Storage](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-getting-started)
* [Getting Started with Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage)







 Next steps 

For more information about using portable storage volumes, see the following tasks:



* [Managing portable storage](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-accessing-portable-storageaccessing-portable-storage)
10. Click Next.
11. Select a file or enter the secret value that you want to associate with the secret.

You must enter the key-value data as a JSON object. The maximum file size is 512 KB.
12. Click Next.
13. Review the details of your secret.
14. Click Add.







 Creating key-value secrets from the CLI 

To create a key-value secret by using the Secrets Manager CLI plug-in, run the [ibmcloud secrets-manager secret-create](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-cli-plugin-secrets-manager-clisecrets-manager-cli-secret-create-command) command. For example, the following command creates a key-value secret and stores {"key1":"value1"} as its value.

ibmcloud secrets-manager secret-create --secret-prototype='{
"name": "example-kv-secret","description": "Description of my key-value secret.","secret_type": "kv","secret_group_id": "67d025e1-0248-418f-83ba-deb0ebfb9b4a","labels": ["dev","us-south"],"data": {"key1": "val1","key2": "val2"},"custom_metadata": {"metadata_custom_key": "metadata_custom_value"},"version_custom_metadata": {"custom_version_key": "custom_version_value"}}'

The command outputs the ID value of the secret, along with other metadata. For more information about the command options, see [ibmcloud secrets-manager secret-create](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-cli-plugin-secrets-manager-clisecrets-manager-cli-secret-create-command).





 Creating key-value secrets with the API
10.242.128.8 sap-app-vsi.saptest.com sap-app-vsi

To prevent the IBM Cloud cloudinit process from reverting the content of /etc/hosts to the previous values on the next restart, change the configuration in /etc/cloud/cloud.cfg. Change manage_etc_host from True to False.

Finally, you need to adapt your storage by creating a file system on the attached storage volume. You can identify the newly attached volume by its size by entering /sbin/fdisk -l and checking the sizes. To safely identify it, find the device ID by clicking Device on the Virtual server instances for VPC page.

Zoom

![Figure 7. Data volumes](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap//images/quickstudy-intel-vs-gen2-image13.png)

Figure 7. Data volumes



1. On the Overview page, check the first 20 digits in Device, and find the same ID under /dev/disk/by-id. Our example device is 07a7-184b....



[root@sap-app-vsi ]# ls -als /dev/disk/by-id/ | grep 07a7-184b4a2f-d768-4 0 lrwxrwxrwx 1 root root 11 May 3 08:30 virtio-07a7-184b4a2f-d786-4 -> ../../vdb

In our example, it's virtio-07a7-184b4a2f-d786-4, which is linked to /dev/vdb.

1. Create a file system on this path:

[root@sap-app-vsi ] mkfs.xfs /dev/vdb



1. Find the related UUID in /dev/disk/by-uuid:
scale_filesystem_block_size File system [block size](https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=considerations-block-size). Spectrum Scale supported block sizes (in bytes) include: 256K, 512K, 1M, 2M, 4M, 8M, 16M. No 4M 
 scale_storage_cluster_filesystem_mountpoint Spectrum Scale storage cluster (owningCluster) file system mount point. The owningCluster is the cluster that owns and serves the file system to be mounted. For more information, see [Mounting a remote GPFS file system](https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=system-mounting-remote-gpfs-file). No /gpfs/fs1 
 scale_storage_cluster_gui_password Password for Spectrum Scale storage cluster GUI. Note: Password should be at least 8 characters, must have one number, one lowercase letter, one uppercase letter, and at least one unique character. Password should not contain username. No None 
 scale_storage_cluster_gui_username GUI user to perform system management and monitoring tasks on storage cluster. Note: Username should be at least 4 characters, any combination of lowercase and uppercase letters. No None 
 scale_storage_image_name Name of the custom image that you would like to use to create virtual machines in your IBM Cloud account to deploy the Spectrum Scale storage cluster. By default, the automation uses a base image plus the Spectrum Scale software and any other software packages that it requires. If you'd like, you can follow the instructions for [Planning for custom images](https://cloud.ibm.com/docs/vpc?topic=vpc-planning-custom-images) to create your own custom image with additional software and use that to build the Spectrum Scale storage cluster through this offering. No hpcc-scale5151-rhel86 
 scale_storage_node_count Total number of storage cluster instances that you need to provision.
If original IOPS/GB for the volume is greater than or equal to 0.3, new IOPS/GB for the volume must also be greater than or equal to 0.3.]

-t, --new-tier
: Endurance Storage Tier (IOPS per GB) [only for endurance volumes]. If no tier is specified, the original tier of the volume is used. Requirements: [If original IOPS/GB for the volume is 0.25, new IOPS/GB for the volume must also be 0.25. If original IOPS/GB for the volume is greater than 0.25, new IOPS/GB for the volume must also be greater than 0.25.]

-f, --force
: Force operation without confirmation.

Examples:

ibmcloud sl file volume-modify 12345678 --new-size 1000 --new-iops 4000

This command modifies volume 12345678 with size of 1000 GB, and IOPS of 4000.

ibmcloud sl file volume-modify 12345678 --new-size 500 --new-tier 4

This command modifies volume 12345678 with size 500 GB, and tier level 4 IOPS per GB.





 ibmcloud sl file volume-options 

List all options for ordering a file storage:

ibmcloud sl file volume-options

Examples:

ibmcloud sl file volume-options

This command lists all options for creating a file storage volume, including storage type, volume size, IOPS, tier level, data center, and snapshot size.





 ibmcloud sl file volume-refresh 

Refresh a dependent duplicate volume with a snapshot from its parent:

ibmcloud sl file volume-refresh VOLUME_ID SNAPSHOT_ID [OPTIONS]

Command options:

-f, --force-refresh
: Force the volume refresh. Ongoing transactions are canceled. --output
: Specify output format. Only JSON is supported now.

Examples:

ibmcloud sl file volume-refresh 123 456

Refresh a dependent duplicate 123 with a snapshot from its parent 456.





 ibmcloud sl file volume-set-note
Store your credentials in a file containing either <access_key>:<secret_key> or :<api_key>. This file needs to have limited access so run:

chmod 0600 <credentials_file>



Now you can mount a bucket using:

s3fs <bucket> <mountpoint> -o url=http{s}://<endpoint> –o passwd_file=<credentials_file>



If the credentials file only has an API key (no HMAC credentials), you'll need to add the ibm_iam_auth flag as well:

s3fs <bucket> <mountpoint> -o url=http{s}://<endpoint> –o passwd_file=<credentials_file> -o ibm_iam_auth



The <bucket> in the example refers to an existing bucket and the <mountpoint> is the local path where you want to mount the bucket. The <endpoint> must correspond to the [bucket's location](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpoints). The credentials_file is the file created with the API key or HMAC credentials.

Now, ls <mountpoint> will list the objects in that bucket as if they were local files (or in the case of object prefixes, as if they were nested directories).





 Performance optimization 

While performance will never be equal to a true local filesystem, it is possible to use some advanced options to increase throughput.

s3fs <bucket_name> <mountpoint> -o url=http{s}://<COS_endpoint> –o passwd_file=<credentials_file> -o cipher_suites=AESGCM -o kernel_cache -o max_background=1000 -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=30 -o multireq_max=30 -o dbglevel=warn





1. cipher_suites=AESGCM is only relevant when using an HTTPS endpoint.
* [net_write_timeout](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-migratingmigrating-config-variables-net-write-timeout)





 innodb_buffer_pool_size_percentage 



* Description: The innodb_buffer_pool_size_percentage value parameter defines your database container's dedicated memory amount.
* Default setting: 50
* Max: 100
* Min: 10
* Requires restart: True



As your database itself uses a given amount of memory, if the innodb_buffer_pool_size_percentage value parameter is configured too high, then your database memory requirements + innodb_buffer_pool_size_percentage can become higher than available memory limits, resulting in an out of memory state (OOM).

The innodb_buffer_pool_size_percentage parameter value differs based on the size of your database. The default value is 50%, which is safe for databases of all sizes. Configure the value as needed; if you encounter OOM then the value is set too high and you should lower it.





 innodb_flush_log_at_trx_commit 



* Description: Controls the balance between strict ACID compliance for commit operations and higher performance
* Default setting: 2



The default setting of 2 is not fully ACID-compliant (Default setting of 1 is required for full ACID compliance), but it is more performant and still is safe.



* Max: 2
* Min: 0
* Requires restart: False



For more information, see [MySQL innodb_flush_log_at_trx_commit documentation](https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.htmlsysvar_innodb_flush_log_at_trx_commit).





 innodb_log_buffer_size 



* Description: The size in bytes of the buffer that InnoDB uses to write to the log files on disk.
* Default setting: 32 MiB
* Max: 4294967295
Get conflicts from replication document 

In the returned JSON, look for the disk_size value. If the value indicates a size of over 1 GB, go to the [IBM Cloud Support portal](https://www.ibm.com/cloud/support) for further advice.

You can check an individual _replicator document for conflicts, as shown in the following example:



* Curl
* Java
* Node
* Python
* Go



curl "$SERVICE_URL/_replicator/$DOCID?conflicts=true"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.GetReplicationDocumentOptions;
import com.ibm.cloud.cloudant.v1.model.ReplicationDocument;

Cloudant service = Cloudant.newInstance();

GetReplicationDocumentOptions replicationDocOptions =
new GetReplicationDocumentOptions.Builder()
.conflicts(true)
.docId("$DOCID")
.build();

ReplicationDocument response =
service.getReplicationDocument(replicationDocOptions).execute()
.getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.getReplicationDocument({
conflicts: true,
docId: '$DOCID'
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.get_replication_document(
conflicts=True,
doc_id='$DOCID'
).get_result()

print(response)
* For file system sizes, see [SAP HANA Storage Requirements](https://www.sap.com/documents/2015/03/74cdb554-5a7c-0010-82c7-eda71af511fa.html?infl=9e705342-b66e-49d2-9023-c7aa7ef6f6af)







 Storage performance considerations 

It is important to calculate your project requirements before you decide on a storage solution. This calculation is critical for selecting network storage because of storage variations and performance considerations.



 Storage impacts on Recovery Time Objective (RTO) of SAP HANA backups 

If you need to restore an SAP HANA system, then the IOPS of your storage has a significant influence on your restore window. Backup windows are not as critical with SAP HANA since all backups are online backups no matter how you configure SAP HANA.

For example, by using IBM Cloud Block Storage for Classic, you can calculate for an approximate 12 TB restore of SAP HANA at a maximum speed. You must create three physical storage devices (block storage iSCSI LUNs) because the maximum size per device is 4 TB. You can create a stripe over these three devices with the Linux® Logical Volume Manager and create one logical device of 12 TB.

The 12 TB provides 3x10 IOPS/GB, which is a total of 122,880 IOPS/GB at 16 KB. This amount gives you a restore time of 1.875 GB per second, or a total restore time of under 2 hours. Since the measurement for the IOPS is taken at a 50/50 distribution of read and write, you can consider the numbers as a lower boundary of restore performance. It is advisable to do backup and restore tests if you rely on a certain restore window.





 Network Block Storage considerations 

The following sections describe the storage considerations that use network block storage for SAP workload scenarios that use various IBM Cloud infrastructure options.



 Network Block or File storage for VMware on Classic Infrastructure 

Using VMware for SAP workloads on IBM Cloud is certified. However it requires choice of the storage and would use the "TDI" delivery model for which you would need to run validation checks to gain SAP Support.