vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Relevant user inputs are expected to get higher confidence, so they are less likely to be considered irrelevant or require clarification
* Irrelevance detection is improved in the presence of direct entity references
* Irrelevance detection is more stable across small changes to input
* Intent detection is more stable regarding occurrence of numerics, such as postal codes
* For German-language assistants, intent detection is more robust in the presence of umlauts



This algorithm was first introduced as the Beta version in June 2022. Since then, support for more languages has been added. This algorithm version was stabilized in December 2022 with minor enhancements since that time.

With this new release, the June 1, 2022 version is now labeled as Previous (01 Jun 2022). The oldest release labeled as 01 Jan 2022 is no longer available for training. As of now, the new Beta version has the same behavior as the Latest (20 Dec 2022) version. Updates to the Beta version will be released soon.

For more information, see [Algorithm version and training](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-algorithm-version).





 10 March 2023 

Dialog session variables now available in Preview
: If you are using dialog in the new experience, you can now see session variables for dialog when debugging in Preview. For more information, see [Variable values in Preview](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-reviewreview-variable-values).





 6 March 2023 

Improvements to algorithm version beta
: Improvements to the current Beta algorithm version include:



* Relevant examples are expected to get higher confidence
* For Spanish-language assistants, intent detection is improved in the presence of direct entity references
* Intent detection is more stable regarding occurrence of numerics, such as postal codes
* Intent detection now accounts for fuzzy closed entity mentions
* For German-language assistants, intent detection is more robust in the presence of umlauts
Training the machine learning model 

In IBM Watson® Knowledge Studio , the creation of the machine learning model involves training the machine learning model and evaluating how well the model performed when annotating test data and blind data.



 Creating a machine learning model 

When you create a machine learning model, you select the document sets that you want to use to train the model and specify the percentage of documents that are to be used as training data, test data, and blind data.



 About this task 

By exploring the performance metrics, you can identify ways to improve the model's accuracy.

> Restriction: Only three machine learning models can be trained at a time per Knowledge Studio instance. If your instance contains multiple workspaces and the number of machine learning models that are being trained in other workspaces totals 3 already, then your request to train the machine learning model in your workspace will be queued until the other training processes are done.





 Procedure 

To create a machine learning model:



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Select Machine Learning Model > Performance.
3. Verify that all of the document sets have been approved and that all annotation conflicts have been resolved through adjudication. Only documents that have become ground truth through adjudication or approval can be used to train the model.
4. Click Train and evaluate.
5. Optional: To specify how you want to allocate documents from your document sets to be used by the system-level training, test, or blind sets, click Edit settings.

See [Document set management](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-improve-mlwks_mamanagedata) for help determining which ratios to apply.
6. Click Train to train the model, or click Train & Evaluate to train the model, evaluate annotations added by the machine learning model, and analyze the performance statistics.

> Important: Training a machine learning model can take several minutes or several hours, depending on the number of human annotations that exist and the total number of words across all documents.
7. Select the document sets that you want to use for training the model.
[Screen capture of correcting a recognized intent](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/correct_intent.png)
4. If the input is unrelated to any of the intents in your application, you can teach your assistant that by selecting the displayed intent, and then clicking Mark as irrelevant.

![Mark as irrelevant screen capture](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/irrelevant.png)

For more information about this action, see [Teaching your assistant about topics to ignore](https://cloud.ibm.com/docs/assistant?topic=assistant-logslogs-mark-irrelevant).



If your intents are not being correctly recognized, consider making the following kinds of changes:



* Add the unrecognized text as an example to the correct intent.
* Move existing examples from one intent to another.
* Consider whether your intents are too similar, and redefine them as appropriate.







 Absolute scoring 

The Watson Assistant service scores each intent’s confidence independently, not in relation to other intents. This approach adds flexibility; multiple intents can be detected in a single user input. It also means that the system might not return an intent at all. If the top intent has a low confidence score (less than 0.2), the top intent is included in the intents array that is returned by the API, but any nodes that condition on the intent are not triggered. If you want to detect the case when no intents with good confidence scores were detected, use the irrelevant special condition in your dialog node. See [Special conditions](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-special-conditions) for more information.

As intent confidence scores change, your dialogs might need restructuring.
Use unrecognized requests to get action recommendations 

IBM Cloud

Plus

Enterprise

Use the Recognition page to analyze unrecognized requests. You can use this information to create new actions that address questions and issues that aren't being answered by your assistant.

You can analyze unrecognized requests in English-language assistants on IBM Cloud only. You need a Plus plan or higher.

Recognition measures the requests within a time period that are recognized and successfully routed to an action. Customer requests are considered unrecognized if:



* The request triggers the No Action Matches action
* The assistant asks a clarifying question and the customer chooses None of the above





 Viewing groups of unrecognized requests 

You can view groups of unrecognized requests for the draft or live environment. Watson Assistant generates groups of similar unrecognized requests from the last 30 days so that you can decide whether to add the requests as example phrases to a new action.

Zoom

![Unrecognized requests](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/analytics-unrecognized-groups.png)

Unrecognized requests

The algorithm that generates groups considers several factors in creating the groups:



* Unrecognized requests that have fewer than 2 or more than 35 significant words are removed from consideration. Common words such as my or is, or punctuation such as ?, are not considered significant. Phrases that are too short or too long are usually not effective as example training phrases for your assistant.
* The unrecognized requests are compared to the latest version of your actions so that requests that would no longer be unrecognized with your latest actions version are filtered out.
* Groups for which the request count is less than 10 are excluded. But if this might result in less than 5 groups, the algorithm tries to produce groups that include more than 5 request counts until a total of 5 groups are produced.
[checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Indicate a preference (bias) in queries ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Review query logging and metrics ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 







 Limit details 

For more information about artifact limits per plan, see the feature documentation:



* [Advanced rules model limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-patternadvanced-rules-limits)
* [Classifier limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-classifierclassifier-limits)
* [Collection limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-limits)
* [Dictionary limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionarydictionary-limits)
Any edits you then make within the dialog skill for the Development assistant will only affect the Development assistant's dialog skill, even though you’re using data from messages sent to the Production assistant.

Similarly, if you create multiple versions of a skill, you might want to use message data from one version to improve the training data of another version.

You cannot access log data from assistants that were created in other service instances.



 Picking a data source 

The term data source refers to the logs compiled from conversations between customers and the assistant or custom application by which a dialog skill was deployed.

When you open the Analytics page, metrics are shown that were generated by user interactions with the current dialog skill. No metrics are shown if the current skill has not been deployed and used by customers.

To populate the metrics with message data from a dialog skill or skill version that was added to a different assistant or custom application, one that has interacted with customers, complete these steps:



1. Click the Data source field to see a list of assistants with log data that you might want to use.

The list includes assistants that have been deployed and to which you have access. Or you can choose to show a list of other deployments. For more information about other types of deployments, see [Show deployment IDs explained](https://cloud.ibm.com/docs/assistant?topic=assistant-logslogs-deployment-id-explained).
2. Choose a data source.



Statistical information for the selected data source is displayed.

Notice the list does not include skill versions. To get data that is associated with a specific skill version, you must know the time frame during which a specific skill version was used by a deployed assistant. You can select the assistant as the data source, and then filter the metrics by the appropriate dates to see only log data that was generated by the assistant while it was using the skill version.





 Show deployment IDs explained 

Applications that use the older v1 runtime API must specify a deployment ID in each messages sent using the /message API. This ID identifies the deployed app that the call was made from. The Analytics page can use this deployment ID to retrieve and display logs that are associated with a specific live application.
5.2.7 Minimize the admission of containers with the NET_RAW capability Not Scored 1 [Pass](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 
 5.2.8 Minimize the admission of containers with added capabilities Not Scored 1 [Pass](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 
 5.2.9 Minimize the admission of containers with capabilities assigned Not Scored 2 [Pass](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 







 5.3 Network policies and CNI 



Section 5.3 Network policies and CNI benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 5.3.1 Ensure that the CNI in use supports Network Policies Not Scored 1 Pass IBM 
 5.3.2 Ensure that all Namespaces have Network Policies defined Scored 2 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 







 5.4 Secrets management 



Section 5.4 Secrets management benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 5.4.1 Prefer using secrets as files over secrets as environment variables Not Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 
 5.4.2 Consider external secret storage Not Scored 2 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 







 5.5 Extensible admission control 



Section 5.5 Extensible admission control benchmark results
[checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Stop words list ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Smart Document Understanding user-trained model ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Smart Document Understanding pretrained model 
 Text classifier enrichment ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Dictionary enrichment ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Entity extractor enrichment ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Machine learning enrichment !
For more information, see [Integrating your custom model with the Discovery tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-integrating-with-wksintegrate-customtooling) in the Discovery v1 documentation.







 Deploying a rule-based model to IBM Watson Natural Language Understanding 

Uploading an advanced rules model to Natural Language Understanding is deprecated. As of June 10, 2021, you will not be able to deploy advanced rules models to Natural Language Understanding.

Deploy the rule-based model to enable an application that uses the Natural Language Understanding service to use the model to find and extract entities that are relevant to your domain.

Attention: This is currently an [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) feature of the service.



 Before you begin 

You must have administrative access to a Natural Language Understanding service instance, and know the IBM Cloud space and instance names that are associated with it.





 Procedure 

To deploy a rule-based model to Natural Language Understanding , complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Rule-based Model > Versions > Rule-based Model tab.
3. Choose the version of the model that you want to deploy.

If there is only one working version of the model, save the current model for deployment by clicking Save for Deployment. This versions the model, which enables you to deploy one version, while you continue to improve the current version. Saving the version might take a few minutes. The option to deploy does not appear until after the version is created.

Note: Each version can be deployed to only one service instance. If you want to deploy the same model to more than one instance, create a version for each instance.
4. Click Deploy, choose to deploy it to Natural Language Understanding, and then click Next.
5. Provide the IBM Cloud space and instance. If necessary, select a different region.
6. Click Deploy.
7. The deployment process might take a few minutes.
New beta next-generation language models and low_latency parameter
: The service now supports a growing number of next-generation language models. The next-generation multimedia and telephony models improve upon the speech recognition capabilities of the service's previous generation of broadband and narrowband models. The new models leverage deep neural networks and bidirectional analysis to achieve both higher throughput and greater transcription accuracy. At this time, the next-generation models support only a limited number of languages and speech recognition features. The supported languages, models, and features will increase with future releases. The next-generation models are beta functionality.

Many of the next-generation models also support a new low_latency parameter that lets you request faster results at the possible expense of reduced transcription quality. When low latency is enabled, the service curtails its analysis of the audio, which can reduce the accuracy of the transcription. This trade-off might be acceptable if your application requires lower response time more than it does the highest possible accuracy.The low_latency parameter is beta functionality.

The low_latency parameter impacts your use of the interim_results parameter with the WebSocket interface. Interim results are available only for those next-generation models that support low latency, and only if both the interim_results and low_latency parameters are set to true.



* For more information about the next-generation models and their capabilities, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).
* For more information about language support for next-generation models and about which next-generation models support low latency, see [Supported next-generation language models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ngmodels-ng-supported).
Lesson 7: Creating a machine learning model 

In this lesson, you will learn how to create a machine learning model in Knowledge Studio.



 About this task 

When you create a machine learning model, you select the document sets that you want to use to train it. You also specify the percentage of documents that are to be used as training data, test data, and blind data. Only documents that became ground truth through approval or adjudication can be used to train the machine learning model.

For more information about the machine learning model, see [Training the machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-train-ml) and [Analyzing machine learning model performance](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-evaluate-ml).





 Procedure 



1. Log in to Knowledge Studio as the administrator.
2. Click Machine Learning Model > Performance > Train and evaluate.
3. Select All, and then click Train & Evaluate.

Training might take more than ten minutes, or even hours, depending on the number of human annotations and the number of words in all the documents.
4. After the machine learning model is trained, you can export it from the Version page, or you can view detailed information about its performance by clicking the Detailed Statistics links that are located above each of the graphs on the Performance page.
5. To view the Training / Test / Blind Sets page, click the Train and evaluate button.
6. To see the documents that human annotators worked on, click View Ground Truth.
7. To see the annotations that the trained machine learning model created on that same set of documents, click View Decoding Results.
8. To view details about the precision, recall, and F1 scores for the machine learning model, click the Performance page.
9. Click the Detailed Statistics links above each of the graphs. On these Statistics pages, you can view the scores for mentions, relations, and coreference chains by using the radio buttons.

You can analyze performance by viewing a summary of statistics for entity types, relation types, and coreference chains.
curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/flac" --data-binary @audio-file.flac "{url}/v1/recognize?model=en-US_Telephony&language_customization_id=636d8494-7e53-436a-8557-30d6b2a63cd7"





 Step 4: Evaluate the results of the next-generation model 

Once you have updated your speech recognition requests to use next-generation models, eliminated unsupported parameters, and re-created any custom language models, you can experiment with speech recognition based on previous- and next-generation models. Compare the resulting transcripts to determine whether the next-generation model produces equivalent or better results. Also consider the performance of requests that use next-generation to determine how much faster you receive results.

You can also compare the word error rate of the previous- and next-generation results. The open-source [Word Error Rate (WER) utility](https://github.com/IBM/watson-stt-wer-python), which is available in Python, can help you measure and compare the accuracy of your results.
[Number of jobs in progress](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-metricsibm_sqlquery_jobs_in_progress) 





 Number of bytes read by completed jobs 

The following table shows the cumulative number of bytes read by all Data Engine jobs that completed successfully.



Table 2. Number of bytes read by completed jobs metric metadata.

 Metadata Description 

 Metric name ibm_sqlquery_bytes_read 
 Metric type gauge 
 Value type byte 
 Segment by Service instance, Service instance name 







 Number of completed jobs 

The following table shows the number of Data Engine jobs that were completed successfully.



Table 3. Number of completed jobs metric metadata.

 Metadata Description 

 Metric name ibm_sqlquery_completed_jobs 
 Metric type gauge 
 Value type none 
 Segment by Service instance, Service instance name 







 Number of failed jobs 

The following table shows the number of Data Engine jobs that failed.



Table 4. Number of failed jobs metric metadata.

 Metadata Description 

 Metric name ibm_sqlquery_failed_jobs 
 Metric type gauge 
 Value type none 
 Segment by Service instance, Service instance name 







 Number of jobs in progress 

The following table shows the number of submitted Data Engine jobs that are in progress (queued or running).



Table 5. Number of jobs in progress metric metadata.

 Metadata Description 

 Metric name ibm_sqlquery_jobs_in_progress 
 Metric type gauge 
 Value type none 
 Segment by Service instance, Service instance name 









 Attributes for segmentation 



 Global attributes 

The following attributes are available for segmenting all of the listed metrics.



Table 6. Global attributes.

 Attribute Attribute name Attribute description 

 Cloud type ibm_ctype The cloud type is a value of public, dedicated, or local. 
 Location ibm_location The location of the monitored resource, which can be a region, a data center, or global. 
 Resource group ibm_resource_group_name The resource group where the service instance was created. 
 Scope ibm_scope The scope is the account, organization, or space GUID associated with this metric.