vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Up to 100 custom dictionaries
* Up to 100 custom pattern extraction models
* Up to 100 custom regular expression models
* Up to 20 custom text classification models
* Up to 20 custom document classification models



Autocompletion, curation, and notice requests are not billed in any plan type.







 How is document pricing calculated? 

IBM uses the maximum number of documents per day and then prorates the document price to calculate the cost for a month.

For example, imagine that 300,000 documents are added to a collection in the first 6 days (6/30) of the month, and then another 1 million documents are added over a two-day span. In the first day (1/30) of the two-day span, 700,000 documents are added. All of the documents (1 million + 300,000 = 1,300,000) remain in the collection index for the rest of the month (23/30). The number of documents for the month might be calculated by using an equation like this:

300K * (6/30) + 700K * (1/30) + 1300K * (23/30)

Document pricing counts the number of indexed documents in each collection. If you reuse data in a second collection, it generates a second set of documents in the index, which are counted separately.

For more information about what counts as a document, see [Document limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-doc-limits).





 What is a custom model? 

Custom models include any of the following model types:



* Discovery entity extractor
* Knowledge Studio machine learning
* Knowledge Studio rule-based



Advanced rules models are not counted as custom models.

Discovery entity extractor models that are trained and published as an enrichment count toward the model limit for the plan. The entity extractor enrichment is what incurs charges, not the workspace or model. The entity extractor enrichment incurs charges whether or not it is applied to a collection.





 Premium 

Premium plans offer developers and organizations a single tenant instance of one or more Watson services for better isolation and security.
Not applicable Machine learning model <br><br> * 1 machine learning model per workspace<br> * 10 model versions per workspace<br> * Maximum number of workspaces is determined by your deployment.<br> * The maximum number of training actions you can perform per month is determined by your deployment.<br><br><br> 
 Publication Export a machine learning model to use for performing text extraction in other Watson applications. Not applicable <br><br> * ZIP file<br><br><br> None 







 Rule-based model 

Table 3: Rule-based model



 Task Typical usage Supported input formats Supported output formats Limits and requirements 

 Rule editor Create or upload documents to the rule editor from which to define classes, regular expressions, and rules. <br><br> * Plain text (added in editor)<br> * CSV file in UTF-8 format<br> * Copied from the All document set<br><br><br> None <br><br> * 1 rule-based model per workspace<br> * 5,000 characters per document<br> * 100 documents per workspace<br> * Maximum size of document title is 256 characters<br> * 200 rules per workspace<br> * 400 classes per workspace<br> * 100 regular expression group per workspace<br> * 100 regular expression entries per regular expression group<br> * 1,000 characters per regular expression entry<br> * 5 rule-based model versions per workspace<br><br><br> 
 Publication Publish a rule-based model to use for performing pattern recognition in other Watson applications. Not applicable <br><br> * PEAR file<br><br><br> Rule-based models can currently be exported to IBM Watson™ Discovery only
The bandwidth for each attached volume is set proportionally based on the [volume size and profile](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profiles).

For optimal bandwidth to be realized, detach and reattach the volume after volume bandwidth allocation in the UI, CLI, or API.



 Adjusting volume bandwidth 

You can change the volume/networking bandwidth ratio to meet your needs, but both volume and network bandwidth must be at least 500 Mbps each. For example, to allow more bandwidth for volumes, you can apportion the previous example in equal allocations:



* Volumes: 2,000 Mbps.
* Network: 2,000 Mbps.



However, before you change the volume/networking bandwidth ratio, be sure to evaluate your instance's network bandwidth requirements. Make sure that the new bandwidth allocation does not have negative effects on your instance’s network performance.

Extending the example for multiple volumes, the bandwidth looks like this:



* Total allocation for volumes is 2,000 Mbps,
* Less boot volume minimum allocation, 393 Mbps,
* Remaining bandwidth for data volumes: 1,607 Mbps.



The volume bandwidth that is available to the instance is apportioned on a per-volume basis. The bandwidth is assigned per volume, not shared between volumes. For example, if you have an instance with four identical attached volumes but you are using only one volume, then that volume can get only the bandwidth that is assigned to it. The volume in use can't access extra bandwidth that is assigned to the unused volumes.





 Unattached volume bandwidth versus attached volume bandwidth 

When you create a stand-alone (unattached) block storage data volume, the volume bandwidth is assigned based on volume capacity, IOPS, and volume profile. The API response for a GET /volume/{id} call shows the bandwidth for an unattached volume like the following example.

{
"active": true,
"bandwidth": 640,
"busy": false,
"capacity": 500,
"created_at": "2021-08-08T06:26:17Z",
[root@hana384-vsi ] vgcreate hana_vg /dev/vdh /dev/vdi /dev/vdj
[root@hana384-vsi ] vgcreate hana_log_vg /dev/vdd /dev/vde /dev/vdf /dev/vdg

Next, three logical volumes need to be defined on top. These logical volumes reflect the file system size requirements for SAP HANA. The following commands are for a 384 GB virtual server:

[root@hana384-vsi ] lvcreate -l 100%VG -i 4 -I 64K -n hana_log_lv hana_log_vg
[root@hana384-vsi ] lvcreate -i 3 -L 384G -I 64K -n hana_shared_lv hana_vg
[root@hana384-vsi ] lvcreate -i 3 -l 100%FREE -I 64K -n hana_data_lv hana_vg

Finally, a file system needs to be created on top of each volume group:

[root@hana384-vsi ] mkfs.xfs /dev/mapper/hana_log_vg-hana_log_lv
[root@hana384-vsi ] mkfs.xfs /dev/mapper/hana_vg-hana_data_lv
[root@hana384-vsi ] mkfs.xfs /dev/mapper/hana_vg-hana_shared_lv

The following entries to /etc/fstab mount the file systems after their mount points (/hana/data, /hana/log and /hana/shared) have been created:

/dev/mapper/hana_log_vg-hana_log_lv /hana/log xfs defaults,swalloc,nobarrier,inode64
/dev/mapper/hana_vg-hana_shared_lv /hana/shared xfs defaults,inode64 0 0
Consider the constraints imposed by the output adapter, as those constraints might relate to the limits defined here. The defined output_limit only relates to how many indexable objects can be sent to the output adapter at once. Once an indexable object is sent to the output adapter, it is "on the clock," as defined by the output_timeout variable. It is possible that the output adapter itself has a throttle preventing it from being able to process as many inputs as it receives. For instance, the orchestration output adapter might have a connection pool, configurable for HTTP connections to the service. If it defaults to 8, for example, and if you set the output_limit to a number greater than 8, then there are processes, on the clock, waiting for a turn to execute. You might then experience timeouts.
* num_threads - The number of parallel threads that can be run at one time. This value can be either an integer, which specifies the number of parallel threads directly, or it can be a string, with the format "xNUM", specifying the multiplication factor of the number of available processors, for example, "x1.5". The default value is "30"









 Configuring service options 

Discovery tells the crawler how to manage crawled files when using IBM Watson™ Discovery.

To access the in-product manual for the discovery-service.conf file, with the most up-to-date information, type the following command from the Crawler installation directory:

man discovery_service.conf

Default options can be changed directly by opening the config/discovery/discovery_service.conf file, and specifying the following values specific to your use case:



* http_timeout - The timeout, in seconds, for the document read/index operation; the default is 125.
* proxy_host_port - (optional) When running the data crawler behind a firewall, you might need to set the proxy hostname and proxy port number in order for the data crawler to talk to Discovery. The default value for this option is an empty string, and if you need to change it, enter the value in the form of "{host}:{port}".
During this process, it collects candidate strings of different lengths.

By default, each model uses a default character_insertion_bias of 0.0. This value is optimized to produce the best balance between hypotheses with different numbers of characters. The default is typically adequate for most speech recognition. However, certain use cases might benefit from favoring hypotheses with shorter or longer strings of characters. In such cases, specifying a change from the default can improve speech recognition.

You can use the character_insertion_bias parameter to indicate that the service is to favor shorter or longer strings as it considers subsequent characters for its hypotheses. The value you provide depends on the characteristics of your audio. The range of acceptable values is from -1.0 to 1.0:



* Negative values cause the service to prefer hypotheses with shorter strings of characters.
* Positive values cause the service to prefer hypotheses with longer strings of characters.



As your value approaches -1.0 or 1.0, the impact of the parameter becomes more pronounced. To determine the most effective value for your scenario, start by setting the value of the parameter to a small increment, such as -0.1, -0.05, 0.05, or 0.1, and assess how the value impacts the transcription results. Then experiment with different values as necessary, adjusting the value by small increments to gauge how much to deviate from the model's default.



 Character insertion bias example 

The following example request directs the service to marginally favor longer strings of character hypotheses by setting the character_insertion_bias parameter to 0.1. This is a small initial increment to determine the effect of the change on transcription results.

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?model=en-US_Telephony&character_insertion_bias=0.1"

IBM Cloud Pak for Data
For example, if you have a worker pool with one worker node per three zones (three total worker nodes) and set the minSize to 4 per zone, the cluster autoscaler does not immediately provision an additional three worker nodes per zone (12 worker nodes total). Instead, the scale-up is triggered by resource requests. If you create a workload that requests the resources of 15 worker nodes, the cluster autoscaler scales up the worker pool to meet this request. Now, the minSize means that the cluster autoscaler does not scale down to fewer than four worker nodes per zone even if you remove the workload that requests the amount. For more information, see the [Kubernetes docs](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.mdwhen-does-cluster-autoscaler-change-the-size-of-a-cluster).
* "maxSize": 2: Specify the maximum number of worker nodes per zone that the cluster autoscaler can scale up the worker pool to. The value must be equal to or greater than the value that you set for the minSize.
* "enabled": false: Set the value to true for the cluster autoscaler to manage scaling for the worker pool. Set the value to false to stop the cluster autoscaler from scaling the worker pool. Later, if you want to [remove the cluster autoscaler](https://cloud.ibm.com/docs/openshift?topic=openshift-cluster-scaling-install-addonautoscaler-remove-console), you must first disable each worker pool in the ConfigMap.



3. Save the configuration file.
4. Get your cluster autoscaler pod.

oc get pods -n kube-system
5. Review the Events section of the cluster autoscaler pod for a ConfigUpdated event to verify that the ConfigMap is successfully updated. The event message for your ConfigMap is in the following format: minSize:maxSize:PoolName:<SUCCESS|FAILED>:error message.
* The number of collections for Continuous Relevancy Training is limited to 5. Because Continuous Relevancy Training works across multiple collections, both query and training times might be extended.
* Continuous Relevancy Training only occurs on top-level fields, and there are limits to how many fields can be used in the training process. In cases where there are more than 10 top level fields, training is more likely to encounter errors.



To use Continuous Relevancy Training:

Once trained, Continuous Relevancy Training is used to influence the results of a natural_language_query when using an environment-level query.

Continuous Relevancy Training can be used at query time by running a multi-collection natural_language_query across all collections in your environment. See [Querying multiple collections](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsmultiple-collections) for instructions.

Continuous Relevancy Training does not affect queries made to a trained or untrained collection-level (/v1/environments/{environment_id}/collections/{collection_id}/query) query call.

Tracking status:

The status of Continuous Relevancy Training can be viewed from the /api/v1/environments endpoint. See the [API reference](https://cloud.ibm.com/apidocs/discoveryget-environment-info). The status provides information about the state of the training operations and informs you if Continuous Relevancy Training is active:

"search_status" : [
{
"scope" : "environment",
"status" : "NO_DATA",
"status_description" : "Discovery is employing a default strategy for document search natural_language_query. Enable query and event logging so we can initiate relevancy training to improve search accuracy.”
}
]
If you have an intent example with an annotation, and another word in that example matches the value or a synonym of the same entity, but the value is not annotated, that omission has impact. The model also learns from the context of the term you did not annotate. Therefore, if you label one term as a mention of an entity in a user example, be sure to label any other applicable mentions also.



1. The Customer_Care_Appointments intent includes two intent examples with the word visit.

![Visit examples intent](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/oe-counter1.png)
2. In the second occurrence of the word, you want to annotate the word visit as an entity value of the @meeting entity. This makes visit equivalent to other @meeting entity values such as appointment, as in I'd like to make an appointment or I'd like to schedule a visit.

![@meeting entity](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/oe-counter2.png)
3. In the first occurrence, the word visit is being used as a verb. It has a different meaning from a meeting. In this case, you can select the word appointment from the intent example, and annotate it as an entity value of the @meeting entity. The model learns from the fact that the word visit in the same example is not annotated.

![Visit unselected](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/oe-counter3.png)









 Enabling system entities
7. Deselect any metadata fields that you want to exclude from the data set for your document classifier to learn from, and then click Next.

Any fields that you include are used as additional features in the classification. All of the fields are selected by default. You might need to scroll horizontally to review all of the fields.
8. On the Classifier page, specify the fields to use for machine learning training and prediction.

Answer field
: Select the field from your training data file with the classification label. From the earlier example, the Label field is the best choice.

Predicted field
: The name of the facet that is generated for the predicted class values. By default, the facet name has the syntax <Answer field value>_predicted. For example, Label_predicted.

Test dataset
: Specifies the data set to use to test the classifier model. By default, the training data CSV file that you uploaded and configured is split into three data sets that are used for training, validation, and test respectively. However, you can optionally specify a separate data set to use for testing the model.

Train federated model
: Creates more than one model, based on values from a specific field in the data set. For example, if the document has a Product field, you can configure the classifier to create a separate classifier model for each product name value that is specified in the field. By default, the classifier creates one machine learning classifier model.

You don't need to specify the field that contains the text to be classified. The system detects this field automatically. You can check which field the analyzable text is extracted from and change it or augment it by changing index type of another field. For more information, see [Identifying the text field](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-edit-collectiontext-field).

Click Next.
9. If you want to apply an enrichment to the text in your training data, select at least one field from the Target fields list where you want to apply enrichments.
All volumes are assigned instance bandwidth proportional to their maximum bandwidth, where the sum of all volume bandwidth equals the overall "volumes" bandwidth.

In our [example](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-bandwidthvolume-adjust-bandwidth), the remaining bandwidth that is allocated on the instance for data volumes is 1,607 Mbps (2000 Mbps less 393 Mbps for the boot volume). After the data volume is attached to the instance, the volume optimally requires 640 Mbps. If this volume is the only attached volume, it gets the full bandwidth allocation. If you had two more volumes of greater capacity, the bandwidth allocation is less.

Unattached volume bandwidth might not be the same as the actual bandwidth that you see after the volume is attached to an instance. The difference is due to the amount of bandwidth that is dedicated to the boot volume and all other attached data volumes.







 Estimating volume bandwidth 

Think about the type of data volume that your workloads require and select the right volume profile. Data intensive workloads might require the higher bandwidth performance of a 10 IOPS/GB profile. For more information about the relationship of volume profiles to compute profiles, see [How virtual server profiles relate to storage profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profiles&interface=uivsi-profiles-relate-to-storage). For more information about how block size affects performance, see [Block storage capacity and performance](https://cloud.ibm.com/docs/vpc?topic=vpc-capacity-performance&interface=uihow-block-size-affects-performance).





 Next steps 

Allocate total volume bandwidth by using the API or CLI.
* The application's presentation layer is a thin layer on a REST API. Therefore, horizontal scaling affects the middle tier more than it affects the presentation tier.







 Determine each tier's subnet size 

The next step is to determine each tier's subnet size (in terms of available addresses). Because each tier of the application has a presence in each zone, each zone requires three subnets.

Consider the following information when you plan each tier's subnet size:



* The database tier (the back end) is the least likely to need dynamic scaling, so these subnets are the smallest. That is, these subnets can contain the least number of available addresses.



* This example uses a /27 CIDR block, which allows for 27 addresses in this tier.



* The middle tier is the most likely to need dynamic scaling, so these subnets are the largest. That is, they must contain the greatest number of available addresses.



* This example uses a /25 CIDR block, which allows for 123 addresses in this tier.



* The front-end tier fits in the middle; it doesn't need as many addresses as the middle tier, but it needs more than the database tier does.



* This example uses a /26 CIDR block, which allows for 59 addresses in this tier.









 Combining the subnets and selecting the address prefixes 

To select an acceptable address prefix for each zone, you need a subnet size that's large enough to accommodate all three of the subnets in each tier, and still leave room for horizontal scaling and future expansion.

A /24 address prefix is the smallest prefix into which these three subnets can be combined (27 + 123 + 59). Select the next larger subnet size, not the smallest. Assigning the next larger subnet size (/23) allows for horizontal scaling beyond the limits that were given previously because it allows for adding new subnets to each layer, from within the same address prefix.

After you determine the correct subnet size, you can assign the actual address prefixes, one for each zone:



Table 2. Zones assigned address prefixes

 Zone Address prefix 

 Zone 1 172.16.0.0/23 
 Zone 2 172.16.2.0/23
While scaling may not directly impact the overall resiliency, it does impact the application availability where the virtual server instance resources are exhausted and are no longer able to handle the increased demand.

Scaling can be done multiple ways. For example, scaling up, where a higher profile is being provisioned from either a snapshot volume or building a new system. Another example, and probably the most effective, is using the Auto scale feature for virtual server instance. Auto scale can add or remove virtual server instances based on the current demand. This automatic scaling helps control cost, where you pay only what you need and delete any virtual server instance that is no longer required. For more information about Auto scale, see [Auto scale for VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-auto-scale-instance-group&interface=uiauto-scale-vpc).





 IBM Cloud load balancer consideration 

Adding a IBM Cloud load balancer provides a single well-known destination and also spreads client requests or loads across 2 or more virtual server instances for better resiliency and improve performance. In addition, IBM Cloud load balancers do health checks to ensure that only "healthy" virtual server instances receive the client request.

IBM Cloud offers two types of load balancer-network load balancer and application load balancer. A network load balancer provides better latency and throughput. Whereas application load balancers distribute traffics based on layer 7 and layer 4 information. See [IBM Cloud load balancer overview](https://cloud.ibm.com/docs/vpc?topic=vpc-nlb-vs-elb) for side-by-side feature comparison to help you choose the right load balancer for your application.





 Backup Consideration 

Snapshots work as backups and are the first line of defense for data protection. While critical applications need a suite of HA and DR solutions, a non-mission critical and non-production system can rely on backup snapshots.

Snapshots can be taken both on bootable and secondary data volumes. If you experience accidental data deletion, you can create new data volumes from the backup snapshots and selected files can be copied over.