C ro A no : A Crowd Annotation Platform for Improving Label Consistency of C hinese NER Dataset (2021)

Abstract: In this paper, we introduce CroAno, a web-based crowd annotation platform for the Chinese named entity recognition (NER). Besides some basic features for crowd annotation like fast tagging and data management, CroAno provides a systematic solution for improving label consistency of Chinese NER dataset. 1) Disagreement Adjudicator: CroAno uses a multi-dimensional highlight mode to visualize instance-level inconsistent entities and makes the revision process user-friendly. 2) Inconsistency Detector: CroAno employs a detector to locate corpus-level label inconsistency and provides users an interface to correct inconsistent entities in batches. 3) Prediction Error Analyzer: We deconstruct the entity prediction error of the model to six fine-grained entity error types. Users can employ this error system to detect corpus-level inconsistency from a model perspective. To validate the effectiveness of our platform, we use CroAno to revise two public datasets. In the two revised datasets, we get an improvement of +1.96% and +2.57% F1 respectively in model performance.

## introduction named entity recognition (ner), the task of detecting and classifying named entities in texts, has made significant progress relying on data-driven methods @xcite . existing supervised approaches to ner require massive high-quality annotated data. since hiring annotation experts is costly and time-consuming, crowd annotation and non-experts annotators are more generally used, the drawback is a higher proportion of inconsistency in the annotation. in this paper, we propose a web-based crowd annotation platform named croano. as shown in figure 1 , croano contains three modules to improve label consistency of the chinese ner dataset. disagreement adjudicator: crowd annotation tools usually distribute the same instance to different annotators, which would cause disagreement. we call this phenomenon instance-level label inconsistency, because this inconsistency occurs in the same instance. instance-level label inconsistency is easy to locate but difficult to display and correct. yedda @xcite employs a comparison report to show these inconsistencies to annotation experts. but it failed to display detailed information of inconsistent entities, and annotation experts cannot directly correct these entities through the comparison report. to solve these display and correction issues, croano uses a multi-dimensional display mode to show inconsistent instances and employs the "click to correct" method to facilitate correction. inconsistency detector: this module is designed to solve corpus-level label inconsistency，especially corpus-level pre&suffix inconsistency. corpus-level pre&suffix inconsistency refers to the inconsistency of whether a descriptive string is included in the entity string. in ontonotes4.0, the string "超过"(over) is included as a prefix of money entity in some instances while in other instances the string is excluded as an external prefix. locating these inconsistent entities is hard because a global perspective is required. croano uses a detector to locate potential inconsistency and provides an interface to support users correct these inconsistent entities in batches. prediction error analyzer: as mentioned above, disagreement adjudicator and inconsistency detector are responsible to solve label inconsistency from a data perspective, while prediction error analyzer can detect label inconsistency from a model perspective. to analyze inconsistency, we deconstruct entity prediction error of the model to a novel error system, which gives entity prediction error more abundant information with six fine-grained error types. croano provides a search api, which supports users to employ the error system to locate specific entities and their context. moreover, croano employs an elaborately designed interface to differentiate the model prediction and the annotation. in summary, the contributions of this paper are as follows: disagreement adjudicator provides a multidimensional highlight mode to visualize and an interface function to revise instance-level inconsistency. inconsistency detector employs locating and revising strategies to correct corpus-level label inconsistency. prediction error analyzer can help algorithm experts detect corpus-level inconsistency from a model perspective. in the two revised datasets, we get an improvement of +1.96% and +2.57% in f1 respectively. it is worth mentioning that the promotion of data can be inherited by any ner model. ## user roles in croano croano defines three roles: the crowd annotator, the annotation expert and the algorithm expert. the crowd annotator completes basic annotation tasks. the annotation expert is responsible for guideline formulation, data management, task distribution, and label consistency optimization. the algorithm expert is responsible for providing analyses and suggestions of the annotation result from the perspective of the model. the crowd annotator uses annotator interface for annotation. after preliminary annotation, results are sent to the annotation expert. the annotation expert uses disagreement adjudicator and inconsistency detector for revision. specifically, the annotation expert uses disagreement adjudicator to modify instance-level inconsistency from multi-annotators, and then uses inconsistency detector to detect and modify corpus-level inconsistency. after that, the promoted dataset is sent to the algorithm expert. the algorithm expert uses prediction error analyzer for evaluation. the algorithm expert can use prediction error analyzer to detect the difference between model prediction and annotators annotation to get promotion direction of annotation or just revise the annotation. the following sections will introduce the main modules of croano. ## annotator interface this section describes the interface, which is designed for easily annotating. as shown in figure 2 , the interface uses different colors to distinguish the entity categories. annotators can locate an entity and its span with the left mouse button. and they can choose an entity category by short-cut key or click the corresponding button in the entity label bar. except for basic annotating function, they can mark the current instance as "annotated" by clicking the first top-left button, filter "annotated instances" by clicking the second top-left button, and get the guideline by clicking the last top-left button. ## disagreement adjudicator this section describes disagreement adjudicator, which is designed for solving instance-level label inconsistency. instance-level inconsistency means disagreement annotations between different annotators within the same instance. apart from annotation errors, different understandings of the guideline can also lead to disagreement. this type of inconsistency needs to be revised by the annotation expert. difficulties in implementing this operation are to visualize disagreement entities and revise. croano uses a novel multi-dimensional highlight mode to visualize disagreement entities. as shown in figure 3 , underlines of text represent annotators annotation, the background represents agreement entities or approved entities. figure 3 shows an instance annotated by two annotators. the "美 国"(america) entity has two black underlines and black background, which means the two annotators consistently annotated this span as gpe. the revision stage is easy to execute. the annotation expert can click any disagreement entity to get a revision dialog. the dialog will display all corresponding disagreement entities, and the annotation expert can select a proper one. ## inconsistency detector this section describes inconsistency detector, which is designed for reducing corpus-level pre&suffix inconsistency. ## prediction error analyzer this section describes prediction error analyzer, which is designed for detecting label consistency from the model perspective. ## experiments in this section, we conduct experiments on a standard benchmark and a medical benchmark to verify the effectiveness of inconsistency detector. ## related works most of the existing crowd annotation tools for ner are dedicated to basic functions such as interface friendliness, the convenience of operation, and annotation prompts, which are all designed for annotators. next, this section will compare the features of croano with the following related works: brat @xcite is a web-based general annotation tool that can handle various annotation tasks, including span annotations and the relationship between spans. as an early annotation tool, brat has great influence. however, compared with crowd annotation platforms such as croano, it cannot manage crowdsource or reduce the label inconsistency. gate @xcite is a web-based collaborative text annotation framework. it enables users to perform complex corpus annotation projects, which involve a distributed team of annotators. same as brat, gate also lacks the ability to reduce the label inconsistency. slate @xcite is a lightweight annotation tool with a terminal-based workflow. it is designed for annotation experts, focusing on fast labeling. slate has a certain visual disagreement adjudication capability. however, due to the lim yedda @xcite provides a systematic solution for text span annotations such as collaborative user annotations, administrator evaluation. yedda can locate the position of disagreement and display it by generating a latex file, but it cannot directly adjudicate disagreement like croano. to the best of our knowledge, croano is the first crowd annotation platform providing the tool that can locate and fix inconsistent labels. croano can also make the full use of the ability of the algorithm expert and the annotation expert. ## conclusion and future directions in this paper, we propose a web-based crowd annotation platform, which provides a systematic solution for improving the label consistency of the chinese ner dataset. to solve instance-level inconsistency, we propose disagreement adjudicator. to solve corpus-level inconsistency, we propose inconsistency detector and prediction error analyzer from statistic and model perspectives respectively. the future directions are to extend croano to a cross-task and multi-language version and implement more prosperous model analysis functions.