vania-janet/multiturn-rag-retrieval-data huggingface dataset:
About this task 

When you deploy the machine learning model, you select the version of it that you want to deploy.





 Procedure 

To deploy a machine learning model to the Natural Language Understanding service, complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select Machine Learning Model > Versions.
3. Choose the version of the model that you want to deploy.

If there is only one working version of the model, create a snapshot of the current model. This versions the model, which enables you to deploy a version, while you continue to improve the current version. The option to deploy does not appear until you create at least one version.

Each version can be deployed to any number of service instances. Each deployed instance of a model version is given a unique Model ID but is identical in all other ways.
4. Click Deploy, choose to deploy it to Natural Language Understanding, and then click Next.
5. Select the IBM Cloud space and instance. If necessary, select a different region.
6. Click Deploy.
7. The deployment process might take a few minutes. To check the status of the deployment, click Status on the Versions tab next to the version that you deployed. If the model is still being deployed, the status indicates "publishing". After deployment completes, the status changes to "available" if the deployment was successful, or "error" if problems occurred.

Once available, make a note of the model ID (model_id). You will provide this ID to the Natural Language Understanding service to enable the service to use your custom model.







 What to do next 

You can list deployed model in the Natural Language Understanding service instance by calling the following API method.

curl --user "apikey:{apikey}" "{url}/v1/models?version=2018-11-16"

Any deployed models will be returned in an array similar to the following one:

{
"models": [
{
"workspace_id": "{workspace_id}",
"version_description": "{version_description}",
* The event_priority in the log section controls the type of events that are sent from the agent
* The default log level is information. This means that only information and higher severity events are transmitted.
* Valid levels are: emergency, alert, critical, error, warning, notice, information, debug and none. Note: The values are listed from high priority to low priority.
* Setting the level to none will block all event collection.







 Logging into a file what metrics are included or excluded 



* Setting metrics_excess_log to true in the log section will enable logging of the custom metrics that are included or excluded.
* Metric logging is disabled by default.
* Logging occurs at INFO-level every 30 seconds and lasts for 10 seconds.
* The metricsfile setting is required to specify the location for the metrics to be written by the agent. The metricsfile.location value is a relative path under the /opt/draios directory. Note: The metricsfile entry is specified at the same level as log( not as a child in the yaml)
* Logging data is formatted as follows:

+/-[type] [metric included/excluded]: metric.name (filter: +/-[metric.filter])



* +/- is a symbol that indicates if the metric is included or excluded. Plus (+) indicates that a metric is included. Minus (-) indicates that a metric is excluded.
* [type] specifies the metric type, for example, statsd.
* [metric included/excluded] indicates in a human readable way whether the metric is included or excluded.
* metric.name indicates the metric name.
* (filter: +/-[metric.filter]) provides information about any filters that are defined in the metrics_filter section in the sysdig-agent-configmap.yaml file.





A sample log entry looks as follows:

-[statsd] metric excluded: mongo.statsd.vsize (filter: -[mongo.statsd.])
+[statsd] metric included: mongo.statsd.netIn (filter: +[mongo.statsd.net])
For example, questions about politics for an assistant that makes pet grooming appointments exclusively.

You teach your assistant about subjects to ignore by marking utterances that discuss subjects which are out of scope for your application as being irrelevant. Such utterances become counterexamples for the model.



The best way to build an assistant that understands your domain and the specific needs of your customers is to take the time to build good training data, especially data that includes counterexamples.

Irrelevance detection is designed to bridge any gaps you might have in your counterexample data as you start to build your skill. An alternative method for evaluating the relevance of a newly submitted utterance is triggered in addition to the standard method.

The supplemental method examines the structure of the new utterance and compares it to the structure of the user example utterances in your training data. This alternate approach helps skills that have few or no counterexamples recognize irrelevant utterances. It is likely to have less of an effect for skills that have a sufficient number of counterexamples defined already.

Note that the new method relies on structural information that is based on data from outside your skill. So, while the new method can be useful as you are starting out, to build an assistant that provides a more customized experience, you want it to use information from data that is derived from within the application's domain. The way to ensure that your assistant does so is by adding your own counterexamples.





 Counterexample limits 

The maximum number of counterexamples that you can create is 25,000.
If you rely on these changed metrics, update accordingly.<br><br><br><br> * From gitVersion to git_version<br> * From gitCommit to git_commit<br> * From gitTreeState to git_tree_state<br> * From buildDate to build_date<br> * From goVersion to go_version<br><br><br> 







 Update after worker nodes 

The following table shows the actions that you must take after you update your worker nodes.



Changes to make after you update the worker nodes to Kubernetes 1.19

 Type Description 

 Deprecated: Beta worker node labels The following beta worker node labels are deprecated and replaced. For now, both sets of labels are supported, but update your workloads to use the new labels, such as in affinity rules for deployments.<br><br><br><br> * From beta.kubernetes.io/os to kubernetes.io/os<br> * From beta.kubernetes.io/arch to kubernetes.io/arch<br> * From failure-domain.beta.kubernetes.io/zone to topology.kubernetes.io/zone<br> * From failure-domain.beta.kubernetes.io/region to topology.kubernetes.io/region<br> * From beta.kubernetes.io/instance-type to node.kubernetes.io/instance-type<br><br><br>
CPU use is driven by the number of requests for metrics (HPAs, kubectl top nodes / pods, and so on) and by API discovery requests. The metrics-server provides a Kubernetes API, so that clients such as kubectl that use API discovery place some load on the metrics-server even if they don't use metrics.

The following symptoms might indicate a need to adjust the metrics-server resources:

- The metrics-server is restarting frequently.

- Deleting a namespace results in the namespace that is stuck in a Terminating state and kubectl describe namespace includes a condition reporting a metrics API discovery error.

- kubectl top pods, kubectl top nodes, other kubectl commands, or applications that use the Kubernetes API to log Kubernetes errors such as:

 {: screen}
The server is currently unable to handle the request (get pods.metrics.k8s.io)

 {: screen}
Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

- HorizontalPodAutoscalers (HPAs) do not scale deployments.

- Running kubectl get apiservices v1beta1.metrics.k8s.io results in a status like:

 {: screen}
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/metrics-server False (FailedDiscoveryCheck) 139d

Show more



 Modify the metrics-server-config config map 

Both CPU and memory have tunable "base" and "per node" settings used to compute a total request.

- baseCPU
- cpuPerNode
- baseMemory
- memoryPerNode

Where:

cpuRequest = baseCPU + cpuPerNode * number_of_nodes
memoryRequest = baseMemory + memoryPerNode * number_of_nodes

{: pre}
Natural Language Understanding 

The installable IBM Watson® Natural Language Understanding service helps you analyze various features of text content at scale.

The Watson packages are deprecated. Please use the [Watson SDKs](https://cloud.ibm.com/docs/watson?topic=watson-using-sdks) to perform Watson related functionality. For more information about which Watson SDK is included in the available runtimes, see the [Functions runtime](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-runtimes) documentation.

Provide text, raw HTML, or a public URL, and Natural Language Understanding gives you results for the features you request. The service cleans HTML content before analysis by default, so the results can ignore most advertisements and other unwanted content. You can create [custom models](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) with Watson Knowledge Studio that can be used to detect custom entities and relations in Natural Language Understanding. For more information about this service, see [Natural Language Understanding](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-getting-started).

The Natural Language Understanding package contains the following entities. For more information, see the Natural Language Understanding API reference by clicking the entity name.



Table 1. National language Understanding package entities

 Entity Type Parameters Description 

 [natural-language-understanding-v1](https://cloud.ibm.com/apidocs/natural-language-understanding) Package username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, Work with the Natural Language Understanding service.
* The confidence scores of the runner-up intents that are detected in the user input are close in value to the confidence score of the top intent.
* The confidence score of the top intent is above 0.2.



Even when these conditions are met, disambiguation does not occur unless two or more independent nodes in your dialog meet the following criteria:



* The node condition includes one of the intents that triggered disambiguation.
* A description of the node's purpose is provided for the node in the node name field. (Alternatively, a description can be included in the external node name field.)



A node with a boolean node condition that evaluates to true is likely to be included in the disambiguation list. For example, if the node checks for an entity type and the entity is mentioned in the user input, it is eligible to be included in the list. Such nodes do not trigger disambiguation, but if disambiguation is triggered, they are likely to be included in the resulting disambiguation list.

Learn more



* [Disambiguation example](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-runtimedialog-runtime-disambig-example)
* [Editing the disambiguation configuration](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-runtimedialog-runtime-disambig-edit)
* [Choosing nodes to disable](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-runtimedialog-runtime-disambig-choose-nodes)
* [Disabling disambiguation](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-runtimedialog-runtime-disambig-disable)
* [Handling none of the above](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-runtimedialog-runtime-handle-none)
As the test documents get used iteratively to improve the model, they can start to influence the model training indirectly. That's why the "blind" set of documents is so important.





 How do I control which documents are allocated to a set? 

When you create a machine learning model, you must specify the ratio of documents from the set to allocate to the train, test, or blind sets. Knowledge Studio automatically applies a ratio of 70/23/7 to the document sets that you use to build a machine learning model. You can change these values.



* To add a set that was annotated by humans to the training set, specify a 100/0/0 breakdown ratio.
* After training with a set, you can use it for testing. To use a document set for testing only, specify a 0/100/0 breakdown ration.
* Only add a document set that has never been used for training or testing to the blind set. To do so, specify a 0/0/100 breakdown ratio for the document set.



Plan to stop using a blind set after a few iterations. The more you use a blind set, the less "blind" it becomes. The same goes for your test data. Rather than discard the sets, you can migrate them to serve a new purpose. Follow this migration flow:

blind --> test --> train

As the sets migrate, you can add new, unseen document sets as blind data, and as test data. But, be sure to retain a method for evaluating accuracy across model versions. One way to establish a baseline is to run an earlier version of the model on the new document sets. The results of this run give you a basis of comparison with runs of newer models on those same sets.
Using autolearning to improve assistant responses 

IBM Cloud

Use autolearning to enable your assistant to learn from interactions with your customers and improve responses.

When customers interact with your assistant, they often make choices. Your assistant can learn from these user decisions.

For example, a customer might ask a question that the assistant isn't sure it understands. The assistant asks a clarifying question so the customer can choose the right action from a list. If customers most often click the same action (option #2, for example), your assistant can learn that option #2 is the best answer.

Next time, the assistant can list option #2 as the first choice, so customers can get to it more quickly. If the pattern persists over time, it can change its behavior even more. Your assistant can return option #2 immediately, rather than asking a clarifying question.

As your assistant learns over time, your customers get the best answer more often and in fewer clicks.



 How autolearning works 

Before your assistant can learn from customer behavior, it must observe a significant amount of real conversation data. The conversations take place in a channel such as the web chat, or in a custom application.

Logs of conversations and user decisions from your live environment are the data source for observation. Your assistant analyzes the logs to gain insights. (It doesn't watch real-time clicks during a conversation.)

When the assistant observes enough real conversation data from the live environment, it gains insights to help improve your assistant, providing a better customer experience.

When you publish your assistant to the live environment, autolearning starts training the assistant.





 Applying autolearning 

You can apply autolearning when the following conditions are met:



* Ask clarifying questions is enabled in global settings. For more information, see [Asking clarifying questions](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-understand-questionsunderstand-questions-ask-clarifying-question).
* You publish a version of your assistant to the live environment.
See also [blind data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_B) and [training data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_T).
* train

The process of setting up a Watson instance with components that enable the system to function in a particular domain (for example: corpus content, training data that generates machine learning models, programmatic algorithms, or other ground truth components) and then making improvements and updates to these components based on accuracy analysis.
* training data

A set of annotated documents that can be used to train machine learning models. See also [blind data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_B) and [testing data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_T).
* true negative

An answer or annotation that is actually incorrect and is predicted to be incorrect.
* true positive

An answer or annotation that is actually correct and is predicted to be correct.
* type system

The type system defines the types of objects that can be discovered in a document. The type system defines all possible entity types and relationships between entity types. You can define any number of different types in a type system. Type systems are generally domain-specific and application-specific.
You can specify only one publicly accessible URL in your API request, therefore you cannot extract sentiment scoring from several URLs. You could, however, compile the text from multiple web pages and then pass that entire compiled text for sentiment analysis.





 Unexpected results from webpage analysis 

Analyzing a webpage might return unexpected results in some cases. To investigate, try setting the return_analyzed_text parameter to true to inspect the actual text that is being analyzed in your request. In cases where [webpage cleaning](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-analyzing-webpageswebpage-cleaning) does not remove enough unwanted text, consider using the [xpath](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-analyzing-webpagesxpath) parameter to focus the analysis on specific elements of the page.





 Explanations for particular results 

Natural Language Understanding does not provide any diagnostic tools to explain why a particular request returns a particular result. The service is designed to provide accurate results for as many text samples as possible, but due to the nature of the machine learning models we use, there is no guarantee that any particular result will look correct from a human perspective.





 Unexpected changes in response and confidence scores 

NLU continuously updates pretrained models and the training algorithms powering custom models to give customers better results. These updates may include differences in overall response and confidence scores. NLU is agnostic to individual use cases and makes these updates to models for the sole purpose of better accuracy and better performance for all customers.

See Section 5.1.2 regarding continuous model updates in our [Terms of Service](https://www.ibm.com/support/customer/csol/terms/?id=i128-0025&lc=endetail-document).
In the latter case, a brief filler or non-lexical utterance, such as "uhm" or "well," can lie between two tokens of the keyword. In this case, previous-generation models can insert a hesitation marker at this position of the transcript. Next-generation models include the actual hesitation in the transcription results. For more information, see [Speech hesitations and hesitation markers](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-responseresponse-hesitation).







 Keyword spotting example 

The following example request sets the keywords parameter to a URL-encoded array of three strings (colorado, tornado, and tornadoes) and the keywords_threshold parameter to 0.5:

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/flac" --data-binary @{path}audio-file.flac "{url}/v1/recognize?keywords=colorado%2Ctornado%2Ctornadoes&keywords_threshold=0.5"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: audio/flac" --data-binary @{path}audio-file.flac "{url}/v1/recognize?keywords=colorado%2Ctornado%2Ctornadoes&keywords_threshold=0.5"

The service finds qualifying occurrences of colorado and tornadoes:

{
"result_index": 0,
"results": [
{
"keywords_result": {
"colorado":
{
"normalized_text": "Colorado",
"start_time": 4.94,
"confidence": 0.91,
"end_time": 5.62
}
],
"tornadoes":
{
"normalized_text": "tornadoes",
"start_time": 1.52,
"confidence": 1.0,
"end_time": 2.15
[2020-12-02](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-release-notesnatural-language-understanding-dec0220) Version 2 Korean entity type system.<br><br>Version 2 Spanish entity type system. 
 [2020-08-01](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-release-notesnatural-language-understanding-aug0120) Taxonomy changes aimed towards standardization of the label names in the default taxonomy. 
 [2019-07-12](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-release-notesnatural-language-understanding-jul1219) <br>Deprecated New English entities model with improved accuracy and confidence scores. August 1, 2021 
 [2019-06-04](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-release-notesnatural-language-understanding-jun0419) <br>Deprecated Fixed a bug that caused entities requests with custom models to ignore the limit option.<br><br>The default limit value for all entities requests is now 50 for all models.<br><br>The maximum limit value of 250 entities has been removed. July 12, 2020 
 [2018-11-16](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-release-notesnatural-language-understanding-nov1618) <br>Deprecated Version 2 Italian entity type system. July 12, 2020 
 [2018-09-21](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-release-notesnatural-language-understanding-sep2118) <br>Deprecated Version 2 Portuguese entity type system. July 12, 2020
Early in the development process, use the Dialog skill analysis for Watson Assistant notebook to help you get started. It offers the following types of insights:



* Examines the terms that are correlated with each intent in your training data to find anomalies that might identify problems that you can investigate further.
* Uses a blind test set that you provide to calculate performance on statistical metrics like Accuracy, Precision, Recall & F1.
* Offers advanced features that you can use to find the causes of common issues such as why some sentences are often misidentified.



To learn more about how this notebook can help you improve your dialog, read this [Medium.com blog post](https://medium.com/ibm-watson/announcing-dialog-skill-analysis-for-watson-assistant-83cdfb968178).
3. After you deploy a version of the assistant, and have some conversation log data collected, run the Measure Watson Assistant Performance notebook.
4. Follow the step-by-step instructions provided with the notebooks to analyze a subset of the dialog exchanges from the logs.

Run the following notebook first:



* Measure: Gathers metrics that focus on coverage (how often the assistant is confident enough to respond to users) and effectiveness (when the assistant does respond, whether the responses are satisfying user needs).



The insights are visualized in ways that make it easier to understand areas for improvement in your assistant.
5. Export a sample set of the logs from ineffective conversations, and then analyze and annotate them.

For example, indicate whether a response is correct. If correct, mark whether it is helpful. If a response is incorrect, then identify the root cause, the wrong intent or entity was detected, for example, or the wrong dialog node was triggered. After identifying the root cause, indicate what the correct choice would have been.
6. Feed the annotated spreadsheet to the Analyze Watson Assistant Effectiveness notebook.



* Effectiveness: Performs a deeper analysis of your logs to help you understand the steps you can take to improve your assistant.



7. Use the Dialog Flow Analysis for Watson Assistant notebook to review your dialog. The notebook can help you pinpoint the dialog nodes where customers most frequently abandon the conversation.