vania-janet/multiturn-rag-retrieval-data huggingface dataset:
In general, aim for a score of .8 out of 1, where 1 means perfect agreement. Because you annotated only two entity types in this tutorial, most of the entity type scores are N/A (not applicable), which means no information is available to give a score.

Figure 1. Reviewing inter-annotator scores with users named dave and phil

![This screen capture shows the inter-annotator scores for a task.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_tutiaa2.png)
4. After you review the scores, you can decide whether you want to approve or reject document sets that are in the SUBMITTED status. Take one of these actions:



* If the scores are acceptable for an annotation set, select the check box and click Accept. Documents that do not overlap with other document sets are promoted to ground truth. Documents that do overlap must first be reviewed through adjudication so that conflicts can be resolved. For this tutorial, accept both document sets.
* If the scores are not acceptable for an annotation set, select the check box and click Reject. The document set needs to be revisited by the human annotator to improve the annotations.









 Results 

When you evaluated the inter-annotator agreement scores, you saw how different pairs of human annotators annotated the same document. If the inter-annotator agreement score was acceptable, you accepted the document set.







 Lesson 6: Adjudicating conflicts in annotated documents 

In this lesson, you will learn how to adjudicate conflicts in documents that overlap between document sets in Knowledge Studio.



 About this task 

When you approve a document set, only the documents that do not overlap with other document sets are promoted to ground truth. If a document is part of the overlap between multiple document sets, you must adjudicate any annotation conflicts before the document can be promoted to ground truth.
Interpreting the scores depends on the complexity of your type system, the amount and complexity of the content annotated, how experienced the human annotators are, and other factors. For example, if the annotation task is focused on labeling parts of speech, you might expect to see a high score because of how well-defined the parts of speech are. But a task that requires deeper analysis of the text, where human interpretation is required, might show lower scores until you take steps to clarify the causes of the ambiguity.

Generally, a type system that includes many entity types and relation types is open to more interpretation, thus inter-annotator agreement might be lower during the early phases of model development. Looking at the scores, you can see which entity types, for example, have low scores. These low scores are an indication that the annotation guidelines need to be improved. By looking at the scores between pairs of human annotators, you can identify whether a particular annotator consistently shows lower scores than others. This annotator might be having problems understanding the guidelines or using the annotation tools, and thus is a candidate for additional training.





 Reviewing inter-annotator agreement scores 

When determining which documents are to be promoted to ground truth, you must review the inter-annotator agreement scores. Documents with low scores are candidates to be rejected and returned to the human annotator for improvement.



 About this task 

When you examine inter-annotator agreement, you examine documents that were annotated by more than one human annotator. If a document is not shared across multiple annotation sets and human annotators, there is no inter-annotator agreement to calculate. When you add annotation sets to a task, ensure that the sets that you want to compare contain the same overlapping documents. You can see which documents are in an annotation set by opening the Assets> Documents page, clicking the Annotation Sets tab, and then clicking the names of the sets.

You might experience situations in which no overlapping documents are found. This might happen, for example, if you create annotation sets in two rounds and add them to the same task. Even though the annotation sets were created at about the same time, they don't have any documents in common.
As the project manager resolves annotation conflicts, the approved annotations are promoted to ground truth.



Keep in mind that human annotation always requires judgment calls. The annotation guidelines can do a lot to ensure that text is correctly and consistently annotated, but even the best guidelines are open to interpretation by humans. To obtain ground truth, you will want to spend time training and educating human annotators so that they can make the best judgments when analyzing domain content.



 Inter-annotator agreement 

After human annotators have annotated the documents, you must determine which documents to promote to ground truth. Start by reviewing the inter-annotator agreement scores. Documents with low scores are candidates to be rejected and returned to the human annotator for improvement.

When calculating inter-annotator agreement scores, the system examines all overlapping documents in all annotation sets in the task, regardless of the status of the sets. Because you cannot accept or reject annotation sets until they are in the Submitted status, you might not want to evaluate inter-annotator agreement until all annotation sets are submitted, or you might want to limit your review to only the human annotators who submitted their completed annotation sets.

The inter-annotator agreement scores show how different human annotators annotated mentions, relations, and coreference chains. You can view the scores by comparing a pair of human annotators (for example, compare all of John's mention annotations to all of Mary's mention annotations). You can also view the scores by comparing specific documents (for example, compare the relation annotations that John made in one document to the relation annotations that Mary made in the same document).

To help you identify areas that require investigation, scores that fall below the value that you specified for the inter-annotator agreement threshold are highlighted in red. In early stages of your annotation project, you might find that relation scores are often worse than mention scores because a perfect relation annotation requires that the mentions that define the relationship be in agreement first.

The score in the All column is a Fleiss Kappa score. It represents how consistently the same annotation was applied by multiple human annotators across all overlapping documents in the task.
A precision score reaches its best value at 1 and worst value at 0. A low precision score indicates that the machine learning model generated incorrect annotations.





 Causes 

Low precision scores can occur for many different reasons that depend on the domain, type system complexity, appropriateness of training documents, human annotator skills, and other factors.





 Resolving the problem 

Tune the performance of your machine learning model by performing one or more of following steps then retrain your model:



1. Identify commonly occurring types with low precision.
2. Identify commonly confused types. This information can be found by looking at the numbers that are off the diagonal in the confusion matrix.
3. Review errors where the machine learning model has high confidence.
4. Find patterns in the false negatives in the confusion matrix.
5. If certain types have low precision scores, review the clarity of the annotation guidelines that apply to those types.



![How to resolve low precision scores.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_lowprec.svg)

Figure 2. How to resolve low precision scores







 Analyzing low recall scores 

Tune the performance of your machine learning model to address low recall scores. At a high level, low recall indicates a need to add more training data.



 Symptoms 

A recall score reaches its best value at 1 and worst value at 0. A low recall score indicates that the machine learning model failed to create annotations that it should have created.





 Causes 

Low recall scores can occur for many different reasons that depend on the domain, type system complexity, appropriateness of training documents, human annotator skills, and other factors.





 Resolving the problem 

Tune the performance of your machine learning model by performing one or more of following steps then retrain your model:



1. Identify commonly occurring types with low recall.
2. Identify commonly confused types. This information can be found by looking at the numbers that are off the diagonal in the confusion matrix.
Another indicator of annotation inconsistency is if you have enough annotations, but the corpus density is low. Density can be impacted when a mention that is significant in the domain literature occurs often, but is annotated as different types across the document set.

Low precision is often an indication that you need to improve annotation consistency. To do so, review the annotation guidelines, better train the human annotators, and ensure that human annotators are working in concert and not in isolation from one another.

Check the inter-annotator agreement score. This score, which measures the degree of agreement between different annotators' output on the same document, is a valuable number. Not only does this score tell you the quality of the ground truth documents that will be used to train the machine learning model, but it also indicates the upper bound of machine learning model performance. A model that is trained on these documents is unlikely to outperform the best agreement that humans can reach. For example, if performance persists at 75 and does not go higher, take a look at the inter-annotator agreement results. If the inter-annotator agreement score is 80, take actions to better train the human annotators and ensure that conflicts are correctly resolved (according to the annotation guidelines) during adjudication. If humans can't agree on how something should be labeled, then it's unlikely that a machine learning model will apply the correct labels.
* Enhance human annotator guidelines

Clear and comprehensive annotator guidelines are a crucial part of a harmonious and successful annotation development effort. Human annotators have a tough job to do. There can be nuances in assigning entity and relation types that are hard to anticipate until you start to work with the domain documents. The guidelines can provide a sanity check to human annotators as they evaluate documents. The guidelines should be a living and changing document, especially at the beginning of the annotation process. They provide a key feedback loop because a human annotator can capture things she learned while annotating a few documents, then as she or someone else annotates a few more documents, new tips and gotchas can be added to the guideline, and so on.
A precision score reaches its best value at 1 and worst value at 0. A low precision score indicates that the machine learning model generated incorrect annotations.





 Causes 

Low precision scores can occur for many different reasons that depend on the domain, type system complexity, appropriateness of training documents, human annotator skills, and other factors.





 Resolving the problem 

Tune the performance of your machine learning model by performing one or more of following steps then retrain your model:



1. Identify commonly occurring types with low precision.
2. Identify commonly confused types. This information can be found by looking at the numbers that are off the diagonal in the confusion matrix.
3. Review errors where the machine learning model has high confidence.
4. Find patterns in the false negatives in the confusion matrix.
5. If certain types have low precision scores, review the clarity of the annotation guidelines that apply to those types.



![How to resolve low precision scores.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_lowprec.svg)

Figure 2. How to resolve low precision scores







 Analyzing low recall scores 

Tune the performance of your machine learning model to address low recall scores. At a high level, low recall indicates a need to add more training data.



 Symptoms 

A recall score reaches its best value at 1 and worst value at 0. A low recall score indicates that the machine learning model failed to create annotations that it should have created.





 Causes 

Low recall scores can occur for many different reasons that depend on the domain, type system complexity, appropriateness of training documents, human annotator skills, and other factors.





 Resolving the problem 

Tune the performance of your machine learning model by performing one or more of following steps then retrain your model:



1. Identify commonly occurring types with low recall.
2. Identify commonly confused types.
Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-410ibm-remediations-and-explanations-410) Shared 
 1.2.24 Ensure that the --audit-log-maxbackup option is set to 10 or as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-410ibm-remediations-and-explanations-410) Shared 
 1.2.25 Ensure that the --audit-log-maxsize option is set to 100 or as appropriate Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-410ibm-remediations-and-explanations-410) Shared 
 1.2.26 Ensure that the --request-timeout option is set as appropriate. Scored 1 Pass IBM 
 1.2.27 Ensure that the --service-account-lookup option is set to true. Scored 1 Pass IBM 
 1.2.28 Ensure that the --service-account-key-file option is set as appropriate. Scored 1 Pass IBM 
 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile options are set as appropriate. Scored 1 Pass IBM 
 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file options are set as appropriate. Scored 1 Pass IBM 
 1.2.31 Ensure that the --client-ca-file option is set as appropriate. Scored 1 Pass IBM 
 1.2.32 Ensure that the --etcd-cafile option is set as appropriate. Scored 1 Pass IBM 
 1.2.33 Ensure that the --encryption-provider-config option is set as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-410ibm-remediations-and-explanations-410) Shared
Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 
 1.2.35 Ensure that the API Server makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 







 1.3 Controller manager 



Section 1.3 Controller manager benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.3.1 Ensure that the --terminated-pod-gc-threshold option is set as appropriate. Scored 1 Pass IBM 
 1.3.2 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.3.3 Ensure that the --use-service-account-credentials option is set to true. Scored 1 Pass IBM 
 1.3.4 Ensure that the --service-account-private-key-file option is set as appropriate. Scored 1 Pass IBM 
 1.3.5 Ensure that the --root-ca-file option is set as appropriate. Scored 1 Pass IBM 
 1.3.6 Ensure that the RotateKubeletServerCertificate option is set to true. Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 1.3.7 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 







 1.4 Scheduler 



Section 1.4 Scheduler benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.4.1 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.4.2 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 









 2 Etcd node configuration 

Review the Etcd Node Configuration results of the version 1.5 CIS Kubernetes benchmark.



Section 2 Etcd Node Configuration benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility
For relatively simple grammars like those in the previous examples, the service can return a result with a low confidence score even when the response does not appear to match the grammar at all. This might seem surprising, but a response with a low confidence score can indicate the best match that the service could find for the given audio, even though the response is unlikely to match the grammar. If the response does not match the grammar, the confidence value of the results is typically low enough to indicate the unlikelihood that the grammar can generate the response.

Always consider the confidence score when evaluating whether a response satisfies a grammar. If you don't know what threshold to set for rejection of a result based on its confidence, use an initial value of 0.5. If you experience unexpectedly large rejection rates of correct results, decrease the confidence threshold; for instance, 0.1 can be a good option for some applications. If you experience large numbers of incorrect recognition results, increase the confidence threshold for your application.

In many cases, an empty result or a result with a very low confidence score is a valid response. It can indicate that the user did not understand the question or the available options. You can always recognize the user's audio response without the grammar, but you then run the risk of getting a result that is not valid for the grammar. Grammars provide more reliable results than n-grams, which always return a result for anything other than complete silence.

Knowing that the result must be one of the valid phrases defined by a grammar is a powerful feature that can simplify an application's response handling. Generally speaking, the service can return results with high confidence only for utterances that are generated by a grammar. For the yesno grammar in the first example to reliably recognize the phrase nope as a valid response, you must add that phrase to the grammar.
curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?word_alternatives_threshold=0.10"

Because of the low threshold, the service returns multiple hypotheses and confidence scores for some of the words, along with the start and end times of all words. The final transcript correctly recognizes the input audio.

{
"result_index": 0,
"results": [
{
"final": true,
"alternatives":
{
"transcript": "yes I ate that tuna ",
"confidence": 0.82
}
],
"word_alternatives":
{
"start_time": 0.0,
"end_time": 0.31,
"alternatives":
{
"word": "yes",
"confidence": 1.0
}
]
},
{
"start_time": 0.31,
"end_time": 0.46,
"alternatives":
{
"word": "I",
"confidence": 1.0
}
]
},
{
"start_time": 0.46,
"end_time": 0.63,
"alternatives":
{
"word": "ate",
"confidence": 0.89
},
{
"word": "eat",
"confidence": 0.11
}
]
},
{
"start_time": 0.63,
"end_time": 0.77,
"alternatives":
{
"word": "that",
"confidence": 0.72
},
{
"word": "the",
"confidence": 0.27
}
]
},
{
Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-119cis-benchmark-remediations-119) Shared 
 1.2.35 Ensure that the API server makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 







 1.3 Controller manager 



Section 1.3 Controller manager benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.3.1 Ensure that the --terminated-pod-gc-threshold option is set as appropriate. Scored 1 Pass IBM 
 1.3.2 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.3.3 Ensure that the --use-service-account-credentials option is set to true. Scored 1 Pass IBM 
 1.3.4 Ensure that the --service-account-private-key-file option is set as appropriate. Scored 1 Pass IBM 
 1.3.5 Ensure that the --root-ca-file option is set as appropriate. Scored 1 Pass IBM 
 1.3.6 Ensure that the RotateKubeletServerCertificate option is set to true. Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-119cis-benchmark-remediations-119) IBM 
 1.3.7 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 







 1.4 Scheduler 



Section 1.4 Scheduler benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.4.1 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.4.2 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 









 2 Etcd node configuration 

Review the Etcd Node Configuration results of the version 1.5 CIS Kubernetes benchmark.



Section 2 Etcd Node Configuration benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility
Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-121cis-benchmark-remediations-121) Shared 
 1.2.35 Ensure that the API server makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 







 1.3 Controller manager 



Section 1.3 Controller manager benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.3.1 Ensure that the --terminated-pod-gc-threshold option is set as appropriate. Scored 1 Pass IBM 
 1.3.2 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.3.3 Ensure that the --use-service-account-credentials option is set to true. Scored 1 Pass IBM 
 1.3.4 Ensure that the --service-account-private-key-file option is set as appropriate. Scored 1 Pass IBM 
 1.3.5 Ensure that the --root-ca-file option is set as appropriate. Scored 1 Pass IBM 
 1.3.6 Ensure that the RotateKubeletServerCertificate option is set to true. Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-121cis-benchmark-remediations-121) IBM 
 1.3.7 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 







 1.4 Scheduler 



Section 1.4 Scheduler benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.4.1 Ensure that the --profiling option is set to false. Scored 1 Pass IBM 
 1.4.2 Ensure that the --bind-address option is set to 127.0.0.1. Scored 1 Pass IBM 









 2 Etcd node configuration 

Review the Etcd Node Configuration results of the version 1.5 CIS Kubernetes benchmark.



Section 2 Etcd Node Configuration benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility