vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you expect to use many load balancers at once, migrate your applications to the paginated API to improve responsiveness and reliability. Contact [IBM support](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar) if you need help migrating your existing client applications.







 25 May 2021 



 For all version dates 

Image from volume. On a POST /images request, you can now specify source_volume with an instance boot volume identity. Specifying the encryption_key property in that request encrypts the image with a root key of your choosing. For details, see [Creating an image from a volume](https://cloud.ibm.com/docs/vpc?topic=vpc-create-ifv&interface=apiimage-from-volume-vpc-api).







 18 May 2021 



 For all version dates 

Snapshots for VPC. Use the new regional snapshot service to create point-in-time copies of your block storage boot or data volumes. Select a snapshot during instance provisioning and restore a new, fully-provisioned boot volume to start the instance. You can also create and attach a data volume from a snapshot within a running virtual server instance. Learn about [creating and using snapshots](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-about) and explore the new [API methods](https://cloud.ibm.com/apidocs/vpcdelete-snapshots).







 6 May 2021 



 For all version dates 

Scheduled scaling. Use scheduled scaling for VPC to schedule actions that automatically add or remove instance group capacity, based on daily, intermittent, or seasonal demand. You can create multiple scheduled actions that scale capacity monthly, weekly, daily, hourly, or even every set number of minutes. Explore the instance group [managers methods](https://cloud.ibm.com/apidocs/vpclist-instance-group-managers) and the new [manager actions methods](https://cloud.ibm.com/apidocs/vpclist-instance-group-manager-actions).
These logical volumes reflect the file system size requirements for SAP HANA. The following commands are for a 256 GB virtual server:

[root@hana256-vsi ] lvcreate -i 3 -I 64K -L 256GB -n hana_log_lv hana_vg
[root@hana256-vsi ] lvcreate -i 3 -I 64K -L 256GB -n hana_shared_lv hana_vg
[root@hana256-vsi ] lvcreate -i 3 -I 64K -l 100%FREE -n hana_data_lv hana_vg

In a 128 GB virtual server, in the example above -L 256GB must be replaced by -L 128GB. These commands will not result in the smallest possible file system size, but they create the smallest configuration, which will fulfill the SAP HANA KPIs. Finally, a file system needs to be created on top of each volume group:

[root@hana256-vsi ] mkfs.xfs /dev/mapper/hana_vg-hana_log_lv
[root@hana256-vsi ] mkfs.xfs /dev/mapper/hana_vg-hana_data_lv
[root@hana256-vsi ] mkfs.xfs /dev/mapper/hana_vg-hana_shared_lv

The following entries to /etc/fstab will mount the file systems after their mount points (/hana/data, /hana/log and /hana/shared) have been created:

/dev/mapper/hana_vg-hana_log_lv /hana/log xfs defaults,swalloc,nobarrier,inode64
/dev/mapper/hana_vg-hana_shared_lv /hana/shared xfs defaults,inode64 0 0
/dev/mapper/hana_vg-hana_data_lv /hana/data xfs defaults,largeio,swalloc,inode64 0 0





 mx2-48x384 profile 

For Virtual Server created based on the mx2-48x384 profile there are:
Top-level fields are fields that are not nested underneath another field. The training only occurs on top-level fields, and there are limits to how many fields can be used in the training process. The more top level fields in a collection, the more training data is required. Also, in cases where there are more than 10 top level fields, training is more likely to encounter errors.







 Error: Training data quality standards not met: You will need additional training queries with labeled examples. (To be considered for training, each example must appear in the top 100 search results for its query.) 



* This error means that you need to add more training data to train successfully. You need at least 49 unique training queries at a minimum, and each one needs at least one rated document. Minimum does not equate to optimal; the size of the collection and other factors can increase the number of training examples needed to meet the minimum.







 Error: Training data quality standards not met: Insufficient number of unique training queries. Expected at least n, but found m. 



* To meet the minimum training requirements, you need at least 49 unique training queries, and each one needs at least one rated document. If you have more than that and are still receiving this error message, check your notices for additional errors.







 Error: Training data quality standards not met: No documents found with non-zero relevance labels. 



* Training data needs enough labeled data that specifies what documents are high value. This means that you need to rate some documents with non-zero values. If using the Discovery tooling, you need to rate some documents as Relevant (10); if using the API, you need to label some document as 1 or above.







 Error: Training data quality standards not met: Training examples have no relevance label variety for X queries. 



* One of the requirements for training is to have sufficient label diversity, meaning that if you want to get a well-trained system, it is recommended that you not only add documents that are the best relevance match, but also documents that are “good” relevant documents. In other words, if you have a scale of 0-4, it helps to have documents rated as 2's and 3's, in addition to those rated as 4's.
For more information about sites and VMware Cloud Director, see [VMware Shared overview](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-shared_overview).

If you need more than 10,000 vApps in a single account, consider the following design options.



* Increase the number of virtual machines in each vApp.
* Use a separate account in that region to host additional vApps.



The number of VMware Shared instances (or virtual data centers) do not impact this limit.







 Related links 



* [Ordering virtual data centers](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-shared_ordering)
* [Viewing and managing virtual data centers](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-shared_viewing-vdc-summary)
* [Operating VMware Shared](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-shared_vcd-ops-guide)
* [VMware Cloud Director](https://www.vmware.com/ca/products/cloud-director.html)
As more documents are involved, it takes longer for each replica to stream the results from disk and across the network. The node that services the request also has much more work to do in combining the results from each database shard.

Overall, the goal is for a view request to require the minimum amount of data from each shard. This practice minimizes the time that the data is in transit and being combined to form the final result. Using the power of views to precompute aggregate data is one way to achieve this aim. This practice reduces the time that your application spends waiting for the request to complete.





In relational databases, normalizing data is often the most efficient way to store data. This practice makes sense when you can use JOIN to easily combine data from multiple tables. You're more likely to need an HTTP GET request for each piece of data with IBM Cloudant. If you reduce the number of requests you need to build a complete picture of a modeled entity, you can present information to your users more quickly.

By using views, you get many of the benefits of normalized data while you maintain the de-normalized version for efficiency.

As an example, in a relational schema, you'd normally represent tags in a separate table and use a connecting table to join tags with their associated documents. This practice allows quick lookup of all documents with a specific tag.

In IBM Cloudant, you'd store tags in a list in each document. You would then use a view to get the documents with a specific tag by [emitting each tag as a key in your view's map function](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreduce). Querying the view for a specific key then provides all the documents with that tag.





It all comes down to the number of HTTP requests that your application makes. There's a cost to opening HTTP connections, particularly HTTPS. While reusing connections helps, making fewer requests overall speeds up the rate that your application can process data.

As a side benefit, when you use de-normalized documents and pre-computed views, you often have the value that your application requires generated ahead of time.
* Configuring search results for uploaded documents: If you are using a collection of uploaded documents and cannot get the correct search results or the results are not concise enough, consider using Smart Document Understanding when you create the data collection.

You can annotate documents based on text formatting. For example, you can teach Discovery that any text in 28-point bold font is a document title. If you apply this information to the collection when you ingest it, you can later use the title field as the source for the title section of your search result.

You can also use Smart Document Understanding to split up large documents into segments that are easier to search. For more information, see the [Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields) topic in the Discovery documentation.
* My response text is surrounded by brackets: If you notice that your response text is surrounded by brackets and quotation marks (["My response text"]) when you test it from the Preview, for example, you might need to change the source field that you're using in the configuration. The unexpected formatting indicates that the value is stored in the source document as an array. Any field that you extract text from must contain a value with a String data type, not an Array data type. When the chat integration shows a response that is extracted from a field that stores the data as an array, it does a straight conversion of the array value into a string, which produces a response that includes the array syntax.

For example, maybe the field in the source document contains an array with a single text value as its only array element:

"title": ["a single array element"]

The array value is converted by the Watson Assistant into this string value:

"title": "["a single array element"]"

As a result, the string is returned in this format in the chat; the surrounding brackets and quotation marks are displayed:

["a single array element"]

If you see this happening, consider choosing a different collection field from which to extract search results.

The Discovery document highlight field stores values in an array.







 Next steps
Each document in the database is stored on a single shard.

Therefore, having many shards enables greater parallelism for any single document request. The reason is that the coordinator sends requests only to the nodes that hold the document. Therefore, if the database has many shards, there are likely to be many other nodes that do not need to respond to the request. These nodes can continue to work on other tasks without interruption from the coordinator request.
2. To respond to a query request, a database must process the results from all the shards.

Therefore, having more shards introduces a greater processing demand. The reason is that the coordinator must make one request per shard, then combine the results before it returns the response to the client.



To help determine a suitable shard count for your database, begin by identifying the most common types of requests that are made by the applications. For example, consider whether the requests are mostly for single document operations, or are the requests mostly queries? Are any of the operations time-sensitive?

For all queries, the coordinator issues read requests to all replicas. This approach is used because each replica maintains its own copy of the indexes that help answer queries. An important consequence of this configuration is that having more shards enables parallel index building if document writes tend to be evenly distributed across the shards in the cluster.

In practice, it is hard to predict the likely indexing load across the nodes in the cluster. Furthermore, predicting indexing load tends to be less useful than addressing request patterns. The reason is that indexing might be required after a document write, but not after a document request. Therefore, considering indexing alone does not provide sufficient information to estimate an appropriate shard count.

When you consider data size, an important consideration is the number of documents per shard. Each shard holds its documents in a large [B-tree](https://en.wikipedia.org/wiki/B-tree) on disk. Indexes are stored in the same way. As more documents are added to a shard, the number of steps that are used to traverse the B-tree during a typical document lookup or query increases. This 'depth increase' tends to slow down requests because more data must be read from caches or disk.
Follow the guidelines in [Sizing SAP HANA - SAP HANA Master Guide](https://help.sap.com/viewer/eb3777d5495d46c5b2fa773206bbfb46/latest/en-US/d4a122a7bb57101493e3f5ca08e6b039.html) to determine the required total storage capacity size for your target SAP HANA system.

The SAP HANA shared volume, and each of the data and log volumes, must be accessible to all nodes (which may be easier to allow network storage access to all nodes within the Subnet used for storage connectivity). There are specific performance criteria that must be met by the attached Network File System (NFS) volumes:



* /hana/data/ and /hana/log volumes, individual volumes are required for each node with a minimum of 10 IOPS/GB
* /hana/shared volume, required to be shared across all nodes with a minimum of 10 IOPS/GB and recommended to increment further to 12 IOPS/GB



For Classic Infrastructure:



* Read [SAP HANA on NetApp FAS Systems with NFS)](https://www.netapp.com/us/media/tr-4290.pdf) to assist configuration of your SAP HANA multi-node system.
* Use the following Network File System (NFS) mount options in /etc/fstab for each volume to mount - rw,bg,hard,timeo=600,intr,noatime,vers=4,minorversion=1,lock,rsize=1048576,wsize=1048576.



After you mount all of your volumes to all the nodes, your multi-node servers are configured and ready to install the SAP HANA multi-node database. Follow the steps in the [SAP HANA Server Installation and Update Guide)](https://help.sap.com/viewer/2c1988d620e04368aa4103bf26f17727/latest/en-US) to install an SAP HANA database of your required version.









 SAP HANA performance
Sizing a cluster is highly dependent on workloads. Here are some general guidelines:

For Spark workloads reading data from IBM Cloud Object Storage, the minimum RAM in a cluster should be at least half the size of the data you want to analyze in any given job. For the best results, the recommended sizing for Spark workloads reading data from the object store is to have the RAM twice the size of the data you want to analyze. If you expect to have a lot of intermediate data, you should size the number of nodes to provide the right amount of HDFS space in the cluster.
* How do I design and size multiple environments for different purposes?

If you want to size multiple environments, for example a production environment with HA, a disaster recovery environment, a staging environment with HA, and a development environment, you need to consider the following aspects.

Each of these environments should use a separate cluster. If you have multiple developers on your team, consider a separate cluster for each developer unless they can share the same cluster credentials. For a development environment, generally, a cluster with 1 master and 2 compute nodes should suffice. For a staging environment where functionality is tested, a cluster with 1 master and 3 compute nodes is recommended. This gives you additional resources to test on a slightly bigger scale before deploying to production. For a disaster recovery environment with more than one cluster, you will need third party remote data replication capabilities.

Because data is persisted in IBM Cloud Object Storage in IBM Analytics Engine, you do not need to have more than one cluster running all the time. If the production cluster goes down, then a new cluster can be spun up using the DevOps tool chain and can be designated as the production cluster. You should use the customization scripts to configure the new cluster exactly like the previous production cluster.
* How is user management done in IBM Analytics Engine?

How do I add more users to my cluster?

All clusters in IBM Analytics Engine are single user, in other words, each cluster has only one Hadoop user ID with which all jobs are executed. User authentication and access control is managed by the IBM Cloud Identity and Access Management (IAM) service.
Kafka needs approximately 1.5 GB per partition for the topic storage.

retention.bytes = 100 MB
log.segment.size = 512 MB
Kafka needs approximately 1 GB per partition for the topic storage.

More storage is needed per partition for indexes. For each log segment, Kafka also stores two indexes. Their sizes are defined by segment.index.size, which is also configurable and has a default of 10 MB. For reference, the storage that is used by indexes is calculated as follows.

2 x number.of.log.segments x segment.index.size

Where

number.of.log.segments = floor(retention.bytes/log.segment.size) + 1





 Managing storage with Event Streams 

When doing topic administration operations, such as creating topics, creating partitions, or changing topic configurations, Event Streams ensures that enough storage is available to satisfy the operation. To do this, for each topic, Event Streams computes the "reserved size" per topic by using the following method.

Reserved size = retention.bytes + log.segment.size + (2 x segment.index.size x number.of.log.segments)

Where

number.of.log.segments = floor(retention.bytes/log.segment.size) + 1

The total reserved storage percentage is also displayed in IBM Cloud Monitoring by the [ibm_eventstreams_instance_reserved_disk_space_percent metric](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-metricsibm_eventstreams_instance_reserved_disk_space_percent).

If there is not enough unreserved storage to satisfy the topic operation, it is rejected with a PolicyViolation error that explains the reason.

The reserved size calculation can change in the future if Kafka storage requirements are updated.





 Examples 

A nonobvious effect is that Kafka can reserve more storage than expected depending on your topic configurations. See the following examples.



1. A topic with retention.bytes of 1 GB, and with a log segment size of 512 MB:

With one partition, it would reserve about 1.5 GB of storage.
Re-upload these document sets and re-create the annotation tasks after the workspace is restored.



2. Understand tokenizer use.

For machine learning models, workspaces use the machine learning-based tokenizer by default. If you are using a dictionary-based tokenizer and have a specific need to continue doing so, you can configure the workspace to use the dictionary-based tokenizer when you restore it. For more information, see [Tokenizers](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-projectwks_tokenizer).
3. Manage model resources.

Your model, its versions, and snapshot data cannot be migrated. The resources (except read-only dictionaries) that you used to train those artifacts can be migrated. Therefore, after the migration, you can re-create the model. The model that is produced performs the same as models that you generated before the migration. They use the same resources for training.

If you have a model that is already deployed and you plan to delete the workspace after you back it up, withdraw the model from deployment. You can rebuild and redeploy the model after you restore the workspace from the backup. For information about undeploying models, see [Undeploying machine learning models](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlpm-um) and [Undeploying rule-based models](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_publishramu-um).

If you fail to withdraw the model from deployment, the result is an orphaned deployed model. Orphaned deployed models continue to generate charges on your monthly bills.
4. Manage read-only dictionary information.

Read-only dictionaries cannot be migrated. Find out where the read-only dictionary was imported from, so you can re-upload it to your workspace after the migration.
5. Make a list of current user roles.
* Review the Size for vSAN cache disks and Number of vSAN cache disks values.





 Enable vSAN deduplication and compression 

vSAN storage depends on the number of servers and your total disk capacity, and the use of deduplication and compression.

The amount of storage reduction from deduplication and compression depends on many factors, including the type of data stored and the number of duplicate blocks. Larger disk groups tend to provide a higher deduplication ratio. For more information, see [Using deduplication and compression](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vsan.doc/GUID-3D2D80CC-444E-454E-9B8B-25C3F620EFED.html).





 Related links 



* [Network interface](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_orderinginstances-network-interface-settings)
* [Procedure to order VMware vSphere instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_orderinginstances-procedure)
* Each segment counts towards the document limit of your plan. Discovery indexes segments, until the plan limit is reached. For document limits, see [Discovery pricing plans](https://cloud.ibm.com/docs/discovery?topic=discovery-discovery-pricing-plans).
* PDF and Word metadata, as well as any custom metadata, is extracted and included in the index with each segment. Every segment of a document includes identical metadata.
* If a split document is updated and needs to be re-uploaded, consider replacing the document, using the [Update document](https://cloud.ibm.com/apidocs/discoveryupdate-a-document) method. Also, consider uploading the document, using the POST method of the /environments/{environment_id}/collections/{collection_id}/documents/{document_id} API, specifying the contents of the parent_id field of one of the current segments as the {document_id} path variable. All segments are overwritten, unless the updated version of the document has fewer total sections than the original. Those older segments remain in the index and can be individually deleted, using the API. For details, see the [API Reference](https://cloud.ibm.com/apidocs/discoverydelete-a-document).







 Importing and exporting models 

After you define a model with the SDU editor, you can save it and reuse it on other collections.

You can import or export your completed SDU model using the toolbar at the top of the editor. Click the last icon and choose Import model or Export model.

Exported models have the file extension of .sdumodel.

An imported model is intended to be used without any further annotations. If you continue annotating after importing it, the model is overwritten. If you plan to import a model into a new collection, it is a good best practice to create a new collection that contains only 1 document, import the model, then upload the remainder of your documents.





 Best practices for annotating documents