vania-janet/multiturn-rag-retrieval-data huggingface dataset:
1.2.9 Ensure that the --authorization-mode argument includes RBAC Scored 1 Pass IBM 
 1.2.10 Ensure that the admission control plugin EventRateLimit is set Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-126cis-benchmark-remediations-126) IBM 
 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set Scored 1 Pass IBM 
 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-126cis-benchmark-remediations-126) IBM 
 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used Not Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-126cis-benchmark-remediations-126) IBM 
 1.2.14 Ensure that the admission control plugin ServiceAccount is set Scored 1 Pass IBM 
 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set Scored 1 Pass IBM 
 1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-126cis-benchmark-remediations-126) IBM 
 1.2.17 Ensure that the admission control plugin NodeRestriction is set Scored 1 Pass IBM 
 1.2.18 Ensure that the --insecure-bind-address argument is not set Scored 1 Pass IBM 
 1.2.19 Ensure that the --insecure-port argument is set to 0 Scored 1 Pass IBM 
 1.2.20 Ensure that the --secure-port argument is not set to 0 Scored 1 Pass IBM 
 1.2.21 Ensure that the --profiling argument is set to false Scored 1 Pass IBM
Section 5.1 RBAC and Service Accounts benchmark results

 Section Recommendation Scored? Level Result Responsibility 

 5.1.1 Ensure that the cluster-admin role is only used where required. Not Scored 1 Pass Shared 
 5.1.2 Minimize access to secrets. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-125cis-benchmark-remediations-125) Shared 
 5.1.3 Minimize wildcard use in Roles and ClusterRoles. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-125cis-benchmark-remediations-125) Shared 
 5.1.4 Minimize access to create pods. Not Scored 1 Pass Shared 
 5.1.5 Ensure that default service accounts are not actively used. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-125cis-benchmark-remediations-125) Shared 
 5.1.6 Ensure that Service Account Tokens are only mounted where necessary. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-125cis-benchmark-remediations-125) Shared 







 5.2 Pod Security Policies 



Section 5.2 Pod Security Policies benchmark results

 Section Recommendation Scored? Level Result Responsibility 

 5.2.1 Minimize the admission of privileged containers. Not Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-125cis-benchmark-remediations-125) Shared 
 5.2.2 Minimize the admission of containers wanting to share the host process ID namespace. Scored 1 [Pass](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-125cis-benchmark-remediations-125) Shared 
 5.2.3 Minimize the admission of containers wanting to share the host IPC namespace.
If the type is unlikely to ever be used in literature from this domain, then remove it from the type system.
* Two types are often interchanged when human annotators annotate documents. Consider whether the two types could be consolidated into one type that accurately represents the concept or relationship. For example, if the type system contains both PERSON and PEOPLE, which are often used interchangeably, it might be best to use one type named PERSONPEOPLE that covers both cases instead of two separate types.



Use caution when you update the type system. If you update it after human annotators have evaluated documentation sets that were associated with the old type system, the human annotators will have to re-evaluate the documentation sets. Be sure that the changes you need to make are significant enough to warrant such rework.
* Investigate further

If your type system, dictionary, and source documents are complete, and human annotation has been done well, but the machine learning model still performs poorly, then something might have gone wrong in the model training process. For example, you should always see high overall scores (over 95%) when testing on the training data.









 Analyzing low F1 scores 

Tune the performance of your machine learning model to address low F1 scores.



 Symptoms 

An F1 score reaches its best value at 1 and worst value at 0. A low F1 score is an indication of both poor precision and poor recall. The machine learning model generates erroneous annotations and fails to find annotations that it should have found.





 Causes 

Low F1 scores can occur for many different reasons that depend on the domain, type system complexity, appropriateness of training documents, human annotator skills, and other factors.





 Resolving the problem 

Tune the performance of your machine learning model by performing one or more of following steps, and then retraining your model:



1. Identify commonly occurring types with low accuracy.

When analyzing relations, look at both the F1 score of the relation type itself and at the F1 score of each of the two entities that participates in the relationship.
2. Identify commonly confused types. This information can be found by looking at the numbers that are off the diagonal in the confusion matrix.
3. Review errors where the machine learning model has high confidence.
4.1.6 Ensure that the kubelet.conf file ownership is set to root:root. Scored 1 Pass IBM 
 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive. Scored 1 Pass IBM 
 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root. Scored 1 Pass IBM 
 4.1.9 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive. Scored 1 Pass IBM 
 4.1.10 Ensure that the kubelet configuration file ownership is set to root:root. Scored 1 Pass IBM 







 4.2 Kubelet 



Section 4.2 Kubelet benchmark results

 Section Recommendation Scored/Not Scored Level Result Responsibility 

 4.2.1 Ensure that the --anonymous-auth option is set to false. Scored 1 Pass IBM 
 4.2.2 Ensure that the --authorization-mode option is not set to AlwaysAllow. Scored 1 Pass IBM 
 4.2.3 Ensure that the --client-ca-file option is set as appropriate. Scored 1 Pass IBM 
 4.2.4 Ensure that the --read-only-port option is set to 0. Scored 1 Pass IBM 
 4.2.5 Ensure that the --streaming-connection-idle-timeout option is not set to 0. Scored 1 Pass IBM 
 4.2.6 Ensure that the --protect-kernel-defaults option is set to true. Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM 
 4.2.7 Ensure that the --make-iptables-util-chains option is set to true Scored 1 Pass IBM 
 4.2.8 Ensure that the --hostname-override option is not set. Not Scored 1 [Fail](https://cloud.ibm.com/docs/containers?topic=containers-cis-benchmark-120cis-benchmark-remediations-120) IBM
In addition, some percentage of documents must occur in all of the annotation sets that you add to the task (you specify the overlap percentage when you create the annotation sets).



 Important 



* An annotation task is a temporal concept that exists to allow human annotators to annotate text in isolated spaces. It also ensures that only approved annotations are promoted to ground truth.
* An annotation set can be included in one active task at a time. To add an annotation set in one task to a different task, you must first delete the task where the annotation set is active.
* If you delete a human annotator's user account, it affects their annotations too. Any annotations in documents that were assigned to that user but were not promoted to ground truth are deleted.
* If the type system or ground truth editor settings change after you create a human annotation task, you must decide whether to propagate the changes to the task. Type system changes can affect annotations; human annotators might need to review and update their documents.
* If the dictionaries change, the changes are not reflected in the current annotation task. To apply resource changes to ground truth, you must create a new annotation task.
* You can have up to 256 annotation tasks per workspace.









 Procedure 

To create an annotation task:



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. Select the Machine Learning Model > Annotations page, then click the Annotation Tasks tab.
3. Click Add Task.
4. Specify a descriptive task name and select the date that the task must be completed.
5. If no annotation sets are available, click Create Annotation Sets.



1. For the base set, select the document set or annotation set that you want to divide into annotation sets.
2. For the overlap value, specify the percentage of documents that you want to include in each annotation set. Inter-annotator agreement scores cannot be calculated unless two or more human annotators annotate the same documents.
You can see which documents are in an annotation set by opening the Assets> Documents page, clicking the Annotation Sets tab, and then clicking the names of the sets.

You might experience situations in which no overlapping documents are found. This might happen, for example, if you create annotation sets in two rounds and add them to the same task. Even though the annotation sets were created at about the same time, they don't have any documents in common. For another example, if you create annotation sets with overlapping documents, but add one annotation set per task instead of adding all of the annotation sets to a single task, no overlapping documents will be found and inter-annotator agreement cannot be calculated.





 Procedure 

To assess annotation agreement between human annotators:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Machine Learning Model > Annotations page, then click the Annotation Tasks tab. Open the task that you want to evaluate.
3. Click Calculate Inter-Annotator Agreement. The default view shows agreement scores for how consistently pairs of human annotators annotated mentions. The top row shows the overall consistency between each pair of annotators, and the table below shows how consistently a pair of annotators labeled specific mentions in the text.
4. To explore how consistently pairs of human annotators annotated relations and coreferences, select Relation or Coreference from the first menu.
5. To explore how consistently a pair of human annotators annotated entities, relations, or coreferences in specific overlapping documents, select Document in the second menu and then select the pair of annotators that you want to evaluate.
6. After reviewing the scores, you can decide whether you want to approve or reject annotation sets that are in Submitted status. After an annotation set is submitted, a check box is displayed next to its name. Take one of these actions:



* If the inter-annotator agreement scores are acceptable for an annotation set, select the check box and click Accept.
Building the ground truth 

The objective of the annotation project is to obtain ground truth, the collection of vetted data that is used to adapt Watson to a particular domain. In Knowledge Studio, human annotators, who are typically experts in the subject matter of the target domain, play a major role in determining ground truth.

A typical workflow includes the following steps:



1. Human annotators submit annotated documents for review.
2. The project manager (who might be a senior domain expert) reviews the accuracy of their work and compares how consistently the human annotators annotated documents that overlap between the annotation sets.
3. If the inter-annotator agreement score is too low, the project manager rejects the annotation set and returns it to the human annotator to be improved. When an annotation set is rejected, all documents in the set are placed back into editable mode.
4. If the project manager approves an annotation set, documents that do not overlap across annotation sets are promoted to ground truth so that the annotations can be used to train the model.
5. The project manager reviews the overlapping documents and resolves the annotation conflicts. At this stage, referred to as adjudication, the team might review and revise the annotation guidelines to help clarify misunderstandings that caused text to be incorrectly or inconsistently annotated by different human annotators.

In some cases, a project manager might want a higher percentage of overlap for evaluating inter-annotator agreement than the percentage of overlap for adjudicating differences. For example, if inter-annotator agreement looks OK, then the project manager might decide that it is OK to promote either annotation to ground truth.
6. As the project manager resolves annotation conflicts, the approved annotations are promoted to ground truth.



Keep in mind that human annotation always requires judgment calls. The annotation guidelines can do a lot to ensure that text is correctly and consistently annotated, but even the best guidelines are open to interpretation by humans. To obtain ground truth, you will want to spend time training and educating human annotators so that they can make the best judgments when analyzing domain content.
To add an annotation set in one task to a different task, you must first delete the task where the annotation set is active.
* If you delete a human annotator's user account, it affects their annotations too. Any annotations in documents that were assigned to that user but were not promoted to ground truth are deleted.
* If the type system or ground truth editor settings change after you create a human annotation task, you must decide whether to propagate the changes to the task. Type system changes can affect annotations; human annotators might need to review and update their documents.
* If the dictionaries change, the changes are not reflected in the current annotation task. To apply resource changes to ground truth, you must create a new annotation task.
* You can have up to 256 annotation tasks per workspace.









 Procedure 

To create an annotation task:



1. Log in as a Knowledge Studio administrator, and select your workspace.
2. Select the Machine Learning Model > Annotations page, then click the Annotation Tasks tab.
3. Click Add Task.
4. Specify a descriptive task name and select the date that the task must be completed.
5. If no annotation sets are available, click Create Annotation Sets.



1. For the base set, select the document set or annotation set that you want to divide into annotation sets.
2. For the overlap value, specify the percentage of documents that you want to include in each annotation set. Inter-annotator agreement scores cannot be calculated unless two or more human annotators annotate the same documents. For example, if you specify a 20% overlap value for a corpus that contains 30 documents, and you divide the corpus into 3 document sets, 6 documents (20%) will be annotated by all human annotators. The remaining 24 documents will be divided among the 3 human annotators (8 each). Thus, each annotator receives 14 documents to annotate (6+8).
* Annotator_1 places a mention on a span of text; Annotator_2 does not.
* Annotator_1's index begins or ends before or after Annotator_2's (there is a partial overlap or subrange of text).
* Annotator_1 assigns an entity type that is different from the entity type that Annotator_2 assigned.



* Relations



* Annotator_1 creates a relation between two mentions; Annotator_2 does not.
* Annotator_1 and Annotator_2 create a relation between the same mentions, but with a different relation type.
* Annotator_1 and Annotator_2 create a relation between the same mentions, but in the reverse order (rare, because the relation between the first mention and second mention is constrained by the type system).



* Coreference chains



* Annotator_1's version of a coreference chain includes (or excludes) mentions that Annotator_2's coreference chain excludes (or includes). The alignment of entities between Annotator_1 and Annotator_2 becomes a matter of scoring.









 Resolving annotation conflicts 

Adjudication is a step that allows you to review annotation conflicts in overlapping documents before you promote the annotations to ground truth. You can compare the annotations added by a pair of human annotators, or compare human annotations to the current ground truth.



 Before you begin 

[Watch](https://www.youtube.com/watch?v=EbexfsuXxoQ&feature=youtu.be) 3-minute video that illustrates how to adjudicate documents.





 About this adjudication task 

After human annotators complete their annotation tasks, they must submit their completed annotation sets for review. When you evaluate the inter-annotator agreement scores, you can see how different pairs of annotators annotated the same document. If the inter-annotator agreement score is acceptable, you approve the annotation set. If a document does not overlap across annotation sets in the task, the annotations in the approved document are promoted to ground truth.
* Annotator_1 places a mention on a span of text; Annotator_2 does not.
* Annotator_1's index begins or ends before or after Annotator_2's (there is a partial overlap or subrange of text).
* Annotator_1 assigns an entity type that is different from the entity type that Annotator_2 assigned.



* Relations



* Annotator_1 creates a relation between two mentions; Annotator_2 does not.
* Annotator_1 and Annotator_2 create a relation between the same mentions, but with a different relation type.
* Annotator_1 and Annotator_2 create a relation between the same mentions, but in the reverse order (rare, because the relation between the first mention and second mention is constrained by the type system).



* Coreference chains



* Annotator_1's version of a coreference chain includes (or excludes) mentions that Annotator_2's coreference chain excludes (or includes). The alignment of entities between Annotator_1 and Annotator_2 becomes a matter of scoring.









 Resolving annotation conflicts 

Adjudication is a step that allows you to review annotation conflicts in overlapping documents before you promote the annotations to ground truth. You can compare the annotations added by a pair of human annotators, or compare human annotations to the current ground truth.



 Before you begin 

[Watch](https://www.youtube.com/watch?v=EbexfsuXxoQ&feature=youtu.be) a 3-minute video that illustrates how to adjudicate documents.





 About this task 

After human annotators complete their annotation tasks, they must submit their completed annotation sets for review. When you evaluate the inter-annotator agreement scores, you can see how different pairs of annotators annotated the same document. If the inter-annotator agreement score is acceptable, you approve the annotation set. If a document does not overlap across annotation sets in the task, the annotations in the approved document are promoted to ground truth.
When you find instances of disagreement, you must decide how to resolve the conflict, either by selecting the correct annotation from those applied by the human annotators or by selecting a different annotation to apply.

A document is available for adjudication when at least one of the following conditions is true:



* The project manager approves two or more annotation sets in a task at the same time, and the same document exists in at least two of the annotation sets (an overlapping document).
* The project manager approves another annotation set before documents in the previously approved annotation sets are adjudicated. If you adjudicate a document that overlaps between annotation set A and annotation set B, promote the annotations to ground truth, and then approve another annotation set, C, that has the same document, the annotations in the newly approved document are promoted automatically to ground truth because conflicts no longer exist. Be aware that the annotations promoted in annotation set C override the ground truth established when the overlapping documents in annotation sets A and B were adjudicated. If you approve annotation set C before you promote the annotations in annotation set A and B, the overlapping documents in set C can be checked for conflicts and adjudicated.



The amount of time that you spend adjudicating might lessen over time if you take time to improve the annotation guidelines. By providing examples and clarifying areas that caused confusion, you can help human annotators learn from their mistakes and prevent future conflicts.

Here are a few examples of various ways that human annotators disagree:



* Mentions



* Annotator_1 places a mention on a span of text; Annotator_2 does not.
* Annotator_1's index begins or ends before or after Annotator_2's (there is a partial overlap or subrange of text).
* Annotator_1 assigns an entity type that is different from the entity type that Annotator_2 assigned.



* Relations



* Annotator_1 creates a relation between two mentions; Annotator_2 does not.
The annotations are isolated from each other and from ground truth. An annotation task is a temporal concept that exists to allow human annotators to annotate text in isolated spaces. In contrast, ground truth is permanent and singular.

![This figures illustrates that the project manager creates annotation sets and assigns them to an annotation task. Dave and Phil, the human annotators, annotate documents in the sets that are assigned to them.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_datatask.svg) Figure 2. This figures illustrates that the project manager creates annotation sets and assigns them to an annotation task. Dave and Phil, the human annotators, annotate documents in the sets that are assigned to them.

After the project manager approves annotation sets in an annotation task, annotations in documents that do not overlap with other annotation sets become ground truth. For documents that overlap between annotation sets (represented by document 2 in this example), the project manager must adjudicate and resolve conflicts. The annotations in overlapping documents do not become ground truth until they are approved through adjudication.

Ground truth is then used for training and testing a machine learning model, or it can be used as the basis for the next iteration of model development. To use ground truth in a new iteration, you must create a new annotation task.

![This figure illustrates how annotations added by two human annotators become ground truth. One document, labeled document 2, is annotated by both human annotators. The annotations in this overlapping document must be adjudicated before they become ground truth.]