vania-janet/multiturn-rag-retrieval-data huggingface dataset:
In addition, Discovery returns a document_retrieval_strategy field that indicates the source of the confidence score:



* untrained
* relevancy_training, or
* continuous_relevancy_training



The document_retrieval_strategy can be used along with the confidence score to determine how the results provided should be used. In cases where load is high, the document_retrieval_strategy returned might be untrained, even if the collection is trained.

confidence can range from 0.0 to 1.0. The higher the number, the more relevant the document.

The confidence score can be found in the query results, under the result_metadata for each document. The document_retrieval_strategy is listed under the retrieval_details.

{
"matching_results": 4,
"session_token": "1_2gYuWJyaWx792Ni4_DQ4C5cbnW",
"retrieval_details" : {
"document_retrieval_strategy" : "relevancy_training",
},
"results": [
{
"id": "eea16dfd5fe6139a25324e7481a32f89_13",
"result_metadata": {
"confidence": 0.5893963975910735,
"score": 0.5006834
}
}
]
}
Show more

For trained collections, the confidence number is calculated based on how relevant the result is estimated to be, compared to the trained model. Trained collections calculate confidence scores only for natural language queries. If you query a trained collection using the Discovery Query Language, or the trained model is temporarily unavailable, the confidence number returned has a document_retrieval_strategy of untrained. The confidence score for a result with the document_retrieval_strategy of untrained is an unsupervised estimate of how relevant the document results are to the query; it is not interchangeable with the the score returned for trained collections. A trained collection can provide better answers to natural language queries than untrained collections.
When increasing the rate, values greater than 100 might produce results that are not discernibly different from 100.

Table 1 provides examples of how the service changes the speaking rate with different values for the rate_percentage parameter. The examples use the en-US_AllisonV3Voice.



Table 1. Using the rate_percentage parameter

 Specification of the rate_percentage parameter Audio sample 

 rate_percentage=0 <br>The service uses the default speaking rate. You can also omit the parameter to get the default behavior. Your browser does not support the audio tag. 
 rate_percentage=-20 <br>The service uses a speaking rate that is 20 percent slower. Your browser does not support the audio tag. 
 rate_percentage=20 <br>The service uses a speaking rate that is 20 percent faster. Your browser does not support the audio tag. 





 Interaction with SSML <prosody> element 

The rate_percentage parameter changes the speaking rate for an entire request. You can also use rate attribute of the the SSML <prosody> element to modify the rate of specific words of text. Although the rate_percentage parameter accepts only a percentage value, the rate attribute accepts values of different forms. For more information, see [The rate attribute](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsprosody-rate).

If you use the rate_percentage parameter and also use the rate attribute of the <prosody> tag to modify the rate for specific text of the same request, the effects are multiplicative. For example, suppose you specify the rate_percentage parameter with a value of -10, which slows the speaking rate by 10 percent. If you also use the <prosody> element to specify an equivalent rate of -10% for specific text of the request, that text is spoken at a rate that is 81 percent of the default rate for the voice.





 Speaking rate examples 

The following HTTP POST /v1/synthesize method uses the rate_percentage query parameter with a value of -5. The resulting audio is spoken at a rate that is five percent slower than it is by default.
Scored 1 Pass IBM 
 1.2.9 Ensure that the --authorization-mode option includes RBAC. Scored 1 Pass IBM 
 1.2.10 Ensure that the admission control plug-in EventRateLimit is set. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 1.2.11 Ensure that the admission control plug-in AlwaysAdmit is not set. Scored 1 Pass IBM 
 1.2.12 Ensure that the admission control plug-in AlwaysPullImages is set. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 1.2.13 Ensure that the admission control plug-in SecurityContextDeny is set if PodSecurityPolicy is not used. Not Scored 1 Pass IBM 
 1.2.14 Ensure that the admission control plug-in ServiceAccount is set. Scored 1 Pass IBM 
 1.2.15 Ensure that the admission control plug-in NamespaceLifecycle is set. Scored 1 Pass IBM 
 1.2.16 Ensure that the admission control plug-in PodSecurityPolicy is set. Scored 1 Pass IBM 
 1.2.17 Ensure that the admission control plug-in NodeRestriction is set. Scored 1 Pass IBM 
 1.2.18 Ensure that the --insecure-bind-address option is not set. Scored 1 Pass IBM 
 1.2.19 Ensure that the --insecure-port option is set to 0. Scored 1 Pass IBM 
 1.2.20 Ensure that the --secure-port option is not set to 0. Scored 1 Pass IBM 
 1.2.21 Ensure that the --profiling option is set to false. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM
1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-49ibm-remediations-and-explanations-49) Shared 
 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-49ibm-remediations-and-explanations-49) Shared 
 1.2.26 Ensure that the --request-timeout argument is set as appropriate. Scored 1 Pass IBM 
 1.2.27 Ensure that the --service-account-lookup argument is set to true. Scored 1 Pass IBM 
 1.2.28 Ensure that the --service-account-key-file argument is set as appropriate. Scored 1 Pass IBM 
 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate. Scored 1 Pass IBM 
 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate. Scored 1 Pass IBM 
 1.2.31 Ensure that the --client-ca-file argument is set as appropriate. Scored 1 Pass IBM 
 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate. Scored 1 Pass IBM 
 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-49ibm-remediations-and-explanations-49) Shared 
 1.2.34 Ensure that encryption providers are appropriately configured.
After human annotators complete their annotation tasks, they must submit their completed annotation sets for review. When you evaluate the inter-annotator agreement scores, you can see how different pairs of annotators annotated the same document. If the inter-annotator agreement score is acceptable, you approve the annotation set. If a document does not overlap across annotation sets in the task, the annotations in the approved document are promoted to ground truth. If a document overlaps across annotation sets, you should adjudicate the document and resolve any annotation conflicts that exist before promoting the annotations to ground truth.

For example, when you adjudicate a document, you might see that one annotator annotated the mention Barack Obama with the entity type PeoplePerson. Another annotator annotated the same string of text as two mentions, assigning the entity type Person to Barack and the entity type Person to Obama. To help ensure proper training of the machine learning model, you should resolve this inconsistency.

Knowledge Studio supports the ability to adjudicate between two annotation sets at a time, or between an annotation set and the current ground truth. If a document overlaps across more than two annotation sets, adjudicate the two annotation sets that you have the greatest confidence in (perhaps because you have greater confidence in the human annotators) to determine ground truth for the document. And then adjudicate the rest of the annotation sets based on the results of the initial adjudication.





 Procedures 

To view overlapping documents and resolve conflicts:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Machine Learning Model > Annotations page. Click the Annotation Tasks tab and open the task that you want to evaluate.
3. Confirm that at least two annotation sets are in In conflict status.
4. Click Check Overlapping Documents for Conflicts. The documents that are in conflict are listed.
5. If you want to ignore the conflicts in an overlapping document and promote the annotations to ground truth without adjudicating them, click Accept.
6.
1.2.16 Ensure that the admission control plug-in PodSecurityPolicy is set Scored 1 [Pass](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) IBM 
 1.2.17 Ensure that the admission control plug-in NodeRestriction is set Scored 1 Pass IBM 
 1.2.18 Ensure that the --insecure-bind-address argument is not set. Scored 1 Pass IBM 
 1.2.19 Ensure that the --insecure-port argument is set to 0. Scored 1 Pass IBM 
 1.2.20 Ensure that the --secure-port argument is not set to 0. Scored 1 Pass IBM 
 1.2.21 Ensure that the --profiling argument is set to false. Scored 1 Pass IBM 
 1.2.22 Ensure that the --audit-log-path argument is set. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) Shared 
 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) Shared 
 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) Shared 
 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate.
Scored 1 Pass IBM 
 1.2.9 Ensure that the --authorization-mode option includes RBAC. Scored 1 Pass IBM 
 1.2.10 Ensure that the admission control plug-in EventRateLimit is set. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-119cis-benchmark-remediations-119) IBM 
 1.2.11 Ensure that the admission control plug-in AlwaysAdmit is not set. Scored 1 Pass IBM 
 1.2.12 Ensure that the admission control plug-in AlwaysPullImages is set. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-119cis-benchmark-remediations-119) IBM 
 1.2.13 Ensure that the admission control plug-in SecurityContextDeny is set if PodSecurityPolicy is not used. Not Scored 1 Pass IBM 
 1.2.14 Ensure that the admission control plug-in ServiceAccount is set. Scored 1 Pass IBM 
 1.2.15 Ensure that the admission control plug-in NamespaceLifecycle is set. Scored 1 Pass IBM 
 1.2.16 Ensure that the admission control plug-in PodSecurityPolicy is set. Scored 1 Pass IBM 
 1.2.17 Ensure that the admission control plug-in NodeRestriction is set. Scored 1 Pass IBM 
 1.2.18 Ensure that the --insecure-bind-address option is not set. Scored 1 Pass IBM 
 1.2.19 Ensure that the --insecure-port option is set to 0. Scored 1 Pass IBM 
 1.2.20 Ensure that the --secure-port option is not set to 0. Scored 1 Pass IBM 
 1.2.21 Ensure that the --profiling option is set to false. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-119cis-benchmark-remediations-119) IBM
<ScoreDistribution value="Iris-versicolor" recordCount="35" confidence="0.486111111111111"/>
<ScoreDistribution value="Iris-virginica" recordCount="37" confidence="0.513888888888889"/>
<Node id="6" score="Iris-versicolor" recordCount="37">
<SimplePredicate field="petal_length" operator="lessThan" value="4.85"/>
<ScoreDistribution value="Iris-setosa" recordCount="0" confidence="0"/>
<ScoreDistribution value="Iris-versicolor" recordCount="34" confidence="0.918918918918919"/>
<ScoreDistribution value="Iris-virginica" recordCount="3" confidence="0.0810810810810811"/>
</Node>
<Node id="7" score="Iris-virginica" recordCount="35">
<SimplePredicate field="petal_length" operator="greaterOrEqual" value="4.85"/>
<ScoreDistribution value="Iris-setosa" recordCount="0" confidence="0"/>
<ScoreDistribution value="Iris-versicolor" recordCount="1" confidence="0.0285714285714286"/>
<ScoreDistribution value="Iris-virginica" recordCount="34" confidence="0.971428571428571"/>
</Node>
</Node>
</Node>
</TreeModel>
</PMML>
Show more







 Using user-defined functions 

User-defined functions work with AQL statements and clauses.

User-defined scalar functions work with AQL statements and clauses, similar to built-in functions.
To view the Training / Test / Blind Sets page, click the Train and evaluate button.
6. To see the documents that human annotators worked on, click View Ground Truth.
7. To see the annotations that the trained machine learning model created on that same set of documents, click View Decoding Results.
8. To view details about the precision, recall, and F1 scores for the machine learning model, click the Performance page.
9. Click the Detailed Statistics links above each of the graphs. On these Statistics pages, you can view the scores for mentions, relations, and coreference chains by using the radio buttons.

You can analyze performance by viewing a summary of statistics for entity types, relation types, and coreference chains. You can also analyze statistics that are presented in a confusion matrix. To see the matrix, change Summary to Confusion Matrix. The confusion matrix helps you compare the annotations that were added by the machine learning model to the annotations in the ground truth.

> Note: In this tutorial, you annotated documents with only a single dictionary for organizations. Therefore, the scores you see are 0 or N/A for most entity types except ORGANIZATION. The numbers are low, but that is expected, because you did not do any human annotation or correction.

Figure 2. Options on the Statistics page for a machine learning model

![This screen capture shows the Statistics page.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_tutanno9.png)
10. Click Versions. On the Versions page, you can take a snapshot of the model and the resources that were used to create it (except for dictionaries and annotation tasks). For example, you might want to take a snapshot before you retrain the model. If the statistics are poorer the next time you train it, you can promote the older version and delete the version that returned poorer results.







 Results
Not Scored 2 Pass IBM 







 3 Control plane configuration 

Review the Control Plane Configuration results of the version 1.5 CIS Kubernetes benchmark.



 3.1 Authentication and authorization 



Section 3.1 Authentication and Authorization benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 3.1.1 Client certificate authentication should not be used for users. Not Scored 2 Pass Shared 







 3.2 Logging 



Section 3.2 Logging benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 3.2.1 Ensure that a minimal audit policy is created. Scored 1 Fail Shared 
 3.2.2 Ensure that the audit policy covers key security concerns. Not Scored 2 Fail Shared 









 4 Worker node security configuration 

Review the Worker Node Security Configuration results of the version 1.5 CIS Kubernetes benchmark.



 4.1 Worker node configuration files 



Section 4.1 Worker Node Configuration benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive. Scored 1 Pass IBM 
 4.1.2 Ensure that the kubelet service file ownership is set to root:root. Scored 1 Pass IBM 
 4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive. Scored 1 Pass IBM 
 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root. Scored 1 Pass IBM 
 4.1.5 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive. Scored 1 Pass IBM 
 4.1.6 Ensure that the kubelet.conf file ownership is set to root:root. Scored 1 Pass IBM 
 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive. Scored 1 Pass IBM 
 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root. Scored 1 Pass IBM 
 4.1.9 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive. Scored 1 Pass IBM
After human annotators complete their annotation tasks, they must submit their completed annotation sets for review. When you evaluate the inter-annotator agreement scores, you can see how different pairs of annotators annotated the same document. If the inter-annotator agreement score is acceptable, you approve the annotation set. If a document does not overlap across annotation sets in the task, the annotations in the approved document are promoted to ground truth. If a document overlaps across annotation sets, you should adjudicate the document and resolve any annotation conflicts that exist before promoting the annotations to ground truth.

For example, when you adjudicate a document, you might see that one annotator annotated the mention Barack Obama with the entity type PeoplePerson. Another annotator annotated the same string of text as two mentions, assigning the entity type Person to Barack and the entity type Person to Obama. To help ensure proper training of the machine learning model, you should resolve this inconsistency.

Knowledge Studio supports the ability to adjudicate between two annotation sets at a time, or between an annotation set and the current ground truth. If a document overlaps across more than two annotation sets, adjudicate the two annotation sets that you have the greatest confidence in (perhaps because you have greater confidence in the human annotators) to determine ground truth for the document. And then adjudicate the rest of the annotation sets based on the results of the initial adjudication.





 Procedure 

To view overlapping documents and resolve conflicts:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Machine Learning Model > Annotations page. Click the Annotation Tasks tab and open the task that you want to evaluate.
3. Confirm that at least two annotation sets are in In conflict status.
4. Click Check Overlapping Documents for Conflicts. The documents that are in conflict are listed.
5. If you want to ignore the conflicts in an overlapping document and promote the annotations to ground truth without adjudicating them, click Accept.
6.
{"begin": float, "end": float, "count": integer},
. . .
],
"non_speech_level": [
{"begin": float, "end": float, "count": integer},
. . .
]
}
}
}
Show more

The audio_metrics field includes an AudioMetrics object that has two fields:



* sampling_interval indicates the interval in seconds (typically 0.1 seconds) at which the service calculated the audio metrics.
* accumulated includes an AudioMetricsDetails object that provides detailed information about the signal characteristics of the input audio.



The following fields of the AudioMetricsDetails object provide unary values:



* final indicates whether the metrics are for the end of the audio stream, meaning that transcription is complete. Currently, the field is always true. The service returns metrics just once per audio stream. The results provide aggregated metrics for the complete stream.
* end_time specifies the end time in seconds of the audio to which the metrics apply. Because the metrics apply to the entire audio stream, the end time is always the length of the audio.
* signal_to_noise_ratio provides the signal-to-noise ratio (SNR) for the audio signal. The value indicates the ratio of speech to noise in the audio. A valid value lies in the range of 0 to 100 decibels (dB). The service omits the field if it cannot compute the SNR for the audio.
* speech_ratio is the ratio of speech to non-speech segments in the audio signal. The value lies in the range of 0.0 to 1.0.
* high_frequency_loss indicates the probability that the audio signal is missing the upper half of its frequency content.



* A value close to 1.0 typically indicates artificially up-sampled audio, which negatively impacts the accuracy of the transcription results.
* A value at or near 0.0 indicates that the audio signal is good and has a full spectrum.
* A value around 0.5 means that detection of the frequency content is unreliable or unavailable.
1.4.1 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.4.2 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 









 2 Etcd node configuration 

Review the Etcd Node Configuration results of the version 1.5 CIS Kubernetes benchmark.



Section 2 Etcd Node Configuration benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 2.1 Ensure that the --cert-file and --key-file options are set as appropriate. Scored 1 Pass IBM 
 2.2 Ensure that the --client-cert-auth option is set to true. Scored 1 Pass IBM 
 2.3 Ensure that the --auto-tls option is not set to true. Scored 1 Pass IBM 
 2.4 Ensure that the --peer-cert-file and --peer-key-file options are set as appropriate. Scored 1 Pass IBM 
 2.5 Ensure that the --peer-client-cert-auth option is set to true. Scored 1 Pass IBM 
 2.6 Ensure that the --peer-auto-tls option is not set to true. Scored 1 Pass IBM 
 2.7 Ensure that a unique Certificate Authority is used for etcd. Not Scored 2 Pass IBM 







 3 Control plane configuration 

Review the Control Plane Configuration results of the version 1.5 CIS Kubernetes benchmark.



 3.1 Authentication and authorization 



Section 3.1 Authentication and Authorization benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 3.1.1 Client certificate authentication should not be used for users. Not Scored 2 Pass Shared 







 3.2 Logging 



Section 3.2 Logging benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 3.2.1 Ensure that a minimal audit policy is created. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-121cis-benchmark-remediations-121) Shared 
 3.2.2 Ensure that the audit policy covers key security concerns.