vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The IAA scores show how different human annotators annotated mentions, relations, and coreference chains. It is a good idea to check IAA scores periodically and verify that human annotators are consistent with each other.

In this tutorial, the human annotators submitted all the document sets for approval. If the inter-annotator agreement scores are acceptable, you can approve the document sets. If you reject a document set, it is returned to the human annotator for improvement.

For more information about inter-annotator agreement, see [Building the ground truth](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-build-groundtruth).





 Procedure 



1. Log in to Knowledge Studio as the administrator and select Machine Learning Model > Annotations. Click the Annotation Tasks tab, then click the Test task.

In the Status column, you can see that the document sets are submitted.
2. Click Calculate Inter-Annotator Agreement.
3. View IAA scores for mention, relations, and coreference chains by clicking the first menu. You can also view agreement by pairs of human annotators. You can also view agreement by specific documents. In general, aim for a score of .8 out of 1, where 1 means perfect agreement. Because you annotated only two entity types in this tutorial, most of the entity type scores are N/A (not applicable), which means no information is available to give a score.

Figure 1. Reviewing inter-annotator scores with users named dave and phil

![This screen capture shows the inter-annotator scores for a task.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_tutiaa2.png)
4. After you review the scores, you can decide whether you want to approve or reject document sets that are in the SUBMITTED status. Take one of these actions:
To compare how different human annotators annotated the same documents, specify an evaluation threshold. If the annotations made by one human annotator differ from the annotations made by another human annotator to the point where the difference results in a low score, it means that the annotators do not agree. The disagreement needs to be investigated and resolved.





 Procedure 

To set the inter-annotator agreement threshold:



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. Select the Settings > IAA Settings tab.
3. Specify a value between 0 and 1, such as .5 or .8, and then click Save.









 Connecting to annotation guidelines 

After you create annotation guidelines for your project, you can configure Knowledge Studio to connect to them. For help with choosing the correct annotation to apply, human annotators can review the guidelines while annotating documents. Administrators can also review the guidelines if they need assistance while resolving annotation conflicts in overlapping documents.



 Procedure 

To connect the ground truth editor and adjudication tool to your annotation guidelines:



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. select the Settings > Annotation Guidelines tab.
3. Specify the URL to where your guidelines are hosted.
4. Click Save. The system connects the ground truth editor and adjudication tool to your annotation guidelines. Depending on the access permissions granted to users when you created the guidelines, human annotators and workspace administrators might be able to update the guidelines after opening them, for example, to add clarifications and examples.







 Annotation guidelines 

There is no prescribed format for how to document the guidelines, but it is important that the guidelines include detailed examples. Human annotators need to understand which entity type to apply to a mention given the context, and know which relation types are valid for a given pair of mentions. Examples drawn from your domain content are often the best way to convey the correct annotation choices to make.

Annotation guidelines are not static.
To determine whether different human annotators are annotating overlapping documents consistently, review the inter-annotator agreement (IAA) scores.

Knowledge Studio calculates IAA scores by examining all overlapping documents in all document sets in the task, regardless of the status of the document sets. The IAA scores show how different human annotators annotated mentions, relations, and coreference chains. It is a good idea to check IAA scores periodically and verify that human annotators are consistent with each other.

In this tutorial, the human annotators submitted all the document sets for approval. If the inter-annotator agreement scores are acceptable, you can approve the document sets. If you reject a document set, it is returned to the human annotator for improvement.

For more information about inter-annotator agreement, see [Building the ground truth](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-build-groundtruth).





 Procedure 



1. Log in to Knowledge Studio as the administrator and select Machine Learning Model > Annotations. Click the Annotation Tasks tab, then click the Test task.

In the Status column, you can see that the document sets are submitted.
2. Click Calculate Inter-Annotator Agreement.
3. View IAA scores for mention, relations, and coreference chains by clicking the first menu. You can also view agreement by pairs of human annotators. You can also view agreement by specific documents. In general, aim for a score of .8 out of 1, where 1 means perfect agreement. Because you annotated only two entity types in this tutorial, most of the entity type scores are N/A (not applicable), which means no information is available to give a score.

Figure 1. Reviewing inter-annotator scores with users named dave and phil

![This screen capture shows the inter-annotator scores for a task.]
This confidence score is not interchangeable with confidence scores that are returned by untrained collections.

The confidence score can range from 0.0 to 1.0. The higher the number, the more relevant the result.

The confidence score can be found in the query results, under the result_metadata for each document. This number is calculated based on how relevant the result is estimated to be, compared to the trained model.

{
"matching_results": 4,
"retrieval_details": {
"document_retrieval_strategy": "trained"
},
"results": [
{
"id": "eea16dfd5fe6139a25324e7481a32f89_13",
"result_metadata": {
"confidence": 0.08793
}
}
]
}

The document_retrieval_strategy can be found under the retrieval_details. If you query a trained collection by using the Discovery Query Language, or the trained model is temporarily disabled, the document_retrieval_strategy is untrained.

For more information on querying a project, see the [Query overview](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-concepts).





 Relevancy training limits 

The following limits apply to relevancy training models:



* One model per project
* 10,000 queries per model
* 40 models per service instance for Enterprise and Premium plans; 20 models for Plus plan instances







 Other ways to improve relevancy 

If you prefer to use the Discovery API to train Discovery, see the [API reference](https://cloud.ibm.com/apidocs/discovery-datalisttrainingqueries).

You also can use the API to add curations. Curations is a beta feature that you can use to teach Discovery to return a specific document every time a certain query is submitted. For more information, see [Curations](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-curations).
Scored 1 Pass IBM 
 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate Scored 1 Pass IBM 
 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) Shared 
 1.2.34 Ensure that encryption providers are appropriately configured Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-411ibm-remediations-and-explanations-411) Shared 
 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers Not Scored 1 Pass IBM 







 1.3 Controller manager 



Section 1.3 Controller manager benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate Scored 1 Pass IBM 
 1.3.2 Ensure that the --profiling argument is set to false Scored 1 Pass IBM 
 1.3.3 Ensure that the --use-service-account-credentials argument is set to true Scored 1 Pass IBM 
 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate Scored 1 Pass IBM 
 1.3.5 Ensure that the --root-ca-file argument is set as appropriate Scored 1 Pass IBM 
 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true Scored 2 Pass IBM 
 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 Scored 1 Pass IBM 







 1.4 Scheduler 



Section 1.4 Scheduler benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.4.1 Ensure that the --profiling argument is set to false Scored 1 Pass IBM 
 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 Scored 1 Pass IBM 









 2 Etcd node configuration
Scored 1 Pass IBM 
 1.2.32 Ensure that the --etcd-cafile argument is set as appropriate Scored 1 Pass IBM 
 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 
 1.2.34 Ensure that encryption providers are appropriately configured Scored 1 [Fail](https://cloud.ibm.com/docs/openshift?topic=openshift-cis-benchmark-412ibm-remediations-and-explanations-412) Shared 
 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers Not Scored 1 Pass IBM 







 1.3 Controller manager 



Section 1.3 Controller manager benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate Scored 1 Pass IBM 
 1.3.2 Ensure that the --profiling argument is set to false Scored 1 Pass IBM 
 1.3.3 Ensure that the --use-service-account-credentials argument is set to true Scored 1 Pass IBM 
 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate Scored 1 Pass IBM 
 1.3.5 Ensure that the --root-ca-file argument is set as appropriate Scored 1 Pass IBM 
 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true Scored 2 Pass IBM 
 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 Scored 1 Pass IBM 







 1.4 Scheduler 



Section 1.4 Scheduler benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.4.1 Ensure that the --profiling argument is set to false Scored 1 Pass IBM 
 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 Scored 1 Pass IBM 









 2 Etcd node configuration
The score says nothing about how many other mentions that were labeled as entity type A by the human annotator were missed by the machine learning model; the recall score reflects that information. See [Analyzing low precision scores](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-evaluate-mlevaluate-mllowp).
* Recall

A measurement that specifies how many mentions that should have been annotated by a given label were actually annotated with that label - the right mentions being those that human annotators identified in the same documents. Recall is determined by the number of correctly labeled annotations divided by the number of annotations that should have been created. A recall score of 1.0 means that every mention that should have been labeled as entity type A was labeled correctly. A low recall score helps you identify places where the machine learning model failed to create an annotation that it should have. The score says nothing about how many other mentions were also labeled as entity type A, but should not have been; the precision score reflects that information. See [Analyzing low recall scores](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-evaluate-mlevaluate-mllowr).
* Percentage of total annotations

A measurement of ground truth that shows how many words were annotated with a given entity type or relation type out of the total number of words that were annotated as any entity type or relation type in the test document set. This statistic is not available for coreferenced mentions. This value can help you to see how prevalent mentions of one type are compared to the other types in your ground truth.
* Percentage of corpus density (by the number of words)

A measurement of ground truth that shows the number of words that were annotated with a given entity type or relation type out of the total number of words, whether annotated or unannotated. This statistic is not available for coreferenced mentions. This value can help you to see how prevalent mentions of this type are compared to all of the other words in your domain documents.
* Percentage of documents that contain the type
Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 4.2.7 Ensure that the --make-iptables-util-chains option is set to true Scored 1 Pass IBM 
 4.2.8 Ensure that the --hostname-override option is not set. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 4.2.9 Ensure that the --event-qps option is set to 0 or a level which ensures appropriate event capture. Not Scored 2 Pass IBM 
 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file options are set as appropriate. Scored 1 Pass IBM 
 4.2.11 Ensure that the --rotate-certificates option is not set to false. Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 4.2.12 Ensure that the RotateKubeletServerCertificate option is set to true. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 4.2.13 Ensure that the kubelet makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 









 5 Kubernetes policies 

Review the Kubernetes Policies results of the version 1.5 CIS Kubernetes benchmark.



 5.1 RBAC and service accounts 



Section 5.1 RBAC and Service Accounts benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 5.1.1 Ensure that the cluster-admin role is only used where required. Not Scored 1 Pass Shared 
 5.1.2 Minimize access to secrets.
See [Analyzing low precision scores](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-evaluate-mlevaluate-mllowp).
* Recall

A measurement that specifies how many mentions that should have been annotated by a given label were actually annotated with that label - the right mentions being those that human annotators identified in the same documents. Recall is determined by the number of correctly labeled annotations divided by the number of annotations that should have been created. A recall score of 1.0 means that every mention that should have been labeled as entity type A was labeled correctly. A low recall score helps you identify places where the machine learning model failed to create an annotation that it should have. The score says nothing about how many other mentions were also labeled as entity type A, but should not have been; the precision score reflects that information. See [Analyzing low recall scores](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-evaluate-mlevaluate-mllowr).
* Percentage of total annotations

A measurement of ground truth that shows how many words were annotated with a given entity type or relation type out of the total number of words that were annotated as any entity type or relation type in the test document set. This statistic is not available for coreferenced mentions. This value can help you to see how prevalent mentions of one type are compared to the other types in your ground truth.
* Percentage of corpus density (by the number of words)

A measurement of ground truth that shows the number of words that were annotated with a given entity type or relation type out of the total number of words, whether annotated or unannotated. This statistic is not available for coreferenced mentions. This value can help you to see how prevalent mentions of this type are compared to all of the other words in your domain documents.
* Percentage of documents that contain the type

A measurement of ground truth that shows how many documents contain a given entity type or relation type. This statistic is not available for coreferenced mentions.
Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-123cis-benchmark-remediations-123) IBM 
 4.2.9 Ensure that the --event-qps option is set to 0 or a level that ensures appropriate event capture. Not Scored 2 Pass IBM 
 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file options are set as appropriate. Scored 1 Pass IBM 
 4.2.11 Ensure that the --rotate-certificates option is not set to false. Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-123cis-benchmark-remediations-123) IBM 
 4.2.12 Ensure that the RotateKubeletServerCertificate option is set to true. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-123cis-benchmark-remediations-123) IBM 
 4.2.13 Ensure that the kubelet makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 









 5 Kubernetes policies 

Review the Kubernetes Policies results of the version 1.5 CIS Kubernetes benchmark.



 5.1 RBAC and service accounts 



Section 5.1 RBAC and Service Accounts benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 5.1.1 Ensure that the cluster-admin role is only used where required. Not Scored 1 Pass Shared 
 5.1.2 Minimize access to secrets. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-123cis-benchmark-remediations-123) Shared 
 5.1.3 Minimize wildcard use in Roles and ClusterRoles. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-123cis-benchmark-remediations-123) Shared
Not Scored 1 Pass Shared 
 5.4.2 Consider external secret storage. Not Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 







 5.5 Extensible admission control 



Section 5.5 Extensible admission control benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller. Not Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 







 5.6 General policies 



Section 5.6 General policies benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 5.6.1 Create administrative boundaries between resources using namespaces. Not Scored 1 Pass Shared 
 5.6.2 Ensure that the seccomp profile is set to docker/default in your pod definitions. Not Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 
 5.6.3 Apply security context to your pods and containers. Not Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 
 5.6.4 The default namespace should not be used. Scored 2 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 









 IBM remediations and explanations 

Review information from IBM on the CIS Benchmark results.



Explanation and remediation

 Section Remediation/Explanation
Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-124cis-benchmark-remediations-124) IBM 
 4.2.9 Ensure that the --event-qps option is set to 0 or a level that ensures appropriate event capture. Not Scored 2 Pass IBM 
 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file options are set as appropriate. Scored 1 Pass IBM 
 4.2.11 Ensure that the --rotate-certificates option is not set to false. Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-124cis-benchmark-remediations-124) IBM 
 4.2.12 Ensure that the RotateKubeletServerCertificate option is set to true. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-124cis-benchmark-remediations-124) IBM 
 4.2.13 Ensure that the kubelet makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 









 5 Kubernetes policies 

Review the Kubernetes Policies results of the version 1.5 CIS Kubernetes benchmark.



 5.1 RBAC and service accounts 



Section 5.1 RBAC and Service Accounts benchmark results

 Section Recommendation Scored? Level Result Responsibility 

 5.1.1 Ensure that the cluster-admin role is only used where required. Not Scored 1 Pass Shared 
 5.1.2 Minimize access to secrets. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-124cis-benchmark-remediations-124) Shared 
 5.1.3 Minimize wildcard use in Roles and ClusterRoles. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-124cis-benchmark-remediations-124) Shared
Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 
 1.2.26 Ensure that the --request-timeout option is set as appropriate. Scored 1 Pass IBM 
 1.2.27 Ensure that the --service-account-lookup option is set to true. Scored 1 Pass IBM 
 1.2.28 Ensure that the --service-account-key-file option is set as appropriate. Scored 1 Pass IBM 
 1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile options are set as appropriate. Scored 1 Pass IBM 
 1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file options are set as appropriate. Scored 1 Pass IBM 
 1.2.31 Ensure that the --client-ca-file option is set as appropriate. Scored 1 Pass IBM 
 1.2.32 Ensure that the --etcd-cafile option is set as appropriate. Scored 1 Pass IBM 
 1.2.33 Ensure that the --encryption-provider-config option is set as appropriate. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 
 1.2.34 Ensure that encryption providers are appropriately configured. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) Shared 
 1.2.35 Ensure that the API Server makes use of only Strong Cryptographic Ciphers. Not Scored 1 Pass IBM 







 1.3 Controller manager 



Section 1.3 Controller manager benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 1.3.1 Ensure that the --terminated-pod-gc-threshold option is set as appropriate. Scored 1 Pass IBM