Tell Me What You Read: Automatic Expertise-Based Annotator Assignment for Text Annotation in Expert Domains (2021)

Abstract: This paper investigates the effectiveness of automatic annotator assignment for text annotation in expert domains. In the task of creating high-quality annotated corpora, expert domains often cover multiple sub-domains (e.g. organic and inorganic chemistry in the chemistry domain) either explicitly or implicitly. Therefore, it is crucial to assign annotators to documents relevant with their fine-grained domain expertise. However, most of existing methods for crowdsoucing estimate reliability of each annotator or annotated instance only after the annotation process. To address the issue, we propose a method to estimate the domain expertise of each annotator before the annotation process using information easily available from the annotators beforehand. We propose two measures to estimate the annotator expertise: an explicit measure using the predefined categories of sub-domains, and an implicit measure using distributed representations of the documents. The experimental results on chemical name annotation tasks show that the annotation accuracy improves when both explicit and implicit measures for annotator assignment are combined.

## introduction preparation of training data has been a critical issue in applying the supervised and semi-supervised methods to real world problems. training data construction is often quite costly and the quality is hard to assure, especially when the task requires expert knowledge of the target domain such as chemistry and medicine. a possible solution to alleviate the lack of annotated data is the use of a crowdsourcing platform. crowdsourced annotation has proven to be successful for lowering the annotation costs in tasks that require general knowledge, such as pos tagging @xcite , textual entailment, word sense disambiguation @xcite , or recognition of general named entities like person and organization @xcite . meanwhile, there is limited success in using crowdsourcing for annotation in expert domains mainly because of the poor annotation quality made by annotators with low domain expertise. in the medical domain, for example, @xcite combine both non-expert crowdsourced annotators and expert annotators for text annotation. we hypothesize that a key challenge of crowdsourced annotation in expert domains lies in the difficulty of estimating the candidate annotators' expertise in terms of the relevance to the documents to be annotated. some crowdsourcing platforms such as amazon mechanical turk foot_0 provide the feature to specify workers that have the expertise in particular domains. however, it is still not optimal for annotation tasks of documents such as scientific papers, as they tend to require expertise in various specific sub-domains. for example, inorganic chemistry and drug discovery fields belong to a broader category of the chemical domain. annotation on drug discovery papers would be difficult for experts in inorganic chemistry because it substantially differs from the drug discovery field. however, limiting the candidate annotators to the experts in drug discovery would result in insufficient number of annotators that is not enough for obtaining a large size annotated corpus within limited time. moreover, it is even hard to identify the required domain knowledge to understand each document with such high granularity. a practical solution to this issue would be to find a way to estimate the relevance between an annotator and a document and to assign annotators to documents based on the relevance between them. it is expected that even if we can not recruit a sufficient number of annotators for each particular sub-domain, we can assign annotators with expertise in different but relevant sub-domains and obtain better quality annotations. this motivated us to investigate the annotator assignment problem for text annotation in expert domains. the main question is whether we can improve the overall text annotation quality by using not only explicit information such as the annotators' major fields of study but also implicit information estimated by other kinds of information available before the annotation process. to answer the question, we conduct a series of experiments on chemical name annotation tasks in academic paper abstracts from various sub-fields of chemistry. the annotators are graduate students in chemistry departments, so they have expertise in different chemistry sub-domains. as the explicit knowledge of the annotators' expertise, we asked the annotators their fields of expertise from predefined categories. we also asked them to submit information about the literature strongly related to their study. we use the literature information to estimate more fine-grained implicit relevance between the annotated document and the annotators that can not be captured by the explicit information. the experimental results indicate that combining both the explicit and implicit information helps estimate the document-annotator relevance and results in better annotation quality. our approach is orthogonal to the annotation aggregation methods @xcite commonly used in the crowdsourced annotation. while the annotation aggregation methods improve the annotation quality by aggregating the annotations from different annotators after the annotation process, our approach improve the annotation by assigning more relevant domain experts to each text before the annotation process. the difference comes from the fact that most of the studies on crowdsouced annotation focus on annotation tasks that require general knowledge or a single expert domain, where the assignment of annotators has relatively small effect on the final annotation quality. thus, it is possible that our approach and the annotation aggregation methods complement each other. our approach is also different from the existing methods to select the best annotators which requires training based on the previous annotation results by the candidate annotators @xcite @xcite . we believe that estimation of the relevance between documents and annotators can also be helpful for such methods as prior knowledge. ## related work preserving the quality of the crowdsourced work is difficult as annotators possess very diverse abilities, skills, or interests @xcite . for this reason, many researchers have proposed their methods for improving the quality of annotations or maintaining it without increasing the cost. some strategies deal with the problem of the ground truth absence -from increasing the number of annotators @xcite , through intelligently weighting them based on their inferred expertise @xcite @xcite to recruiting a small group of top quality workers @xcite @xcite or seeing how they learn over time @xcite . when it comes to methods used for selecting the best annotators, @xcite utilize a sequential bayesian estimation algorithm for continuous tracking and selecting the best annotators over time. markov decision process can be used to model agent conduct during the consensus tasks and to predict a candidate annotator's work @xcite . @xcite introduce a recruiting algorithm based on a variation of the multi-armed bandit model (mab) outperforming previous methods by up to a remarkable 300%. recommendation of tasks to workers has also been studied with use of a worker's task browsing history @xcite , or taking into account implicit negative feedback @xcite . however, all these approaches focus on annotators, not the target data. unlike these methods, our approach is to assign documents to experts in potentially different sub-domains so that their expert domains are as close as possible to those of the assigned documents. an example of annotation improvement in a scenario requiring specialized expertise (clinical nlp) is given in @xcite , where authors pay attention to documents being annotated. the main problem they aim to solve is not only the lack of ground truth for training and benchmarking but also ambiguity in the target documents. they have shown that, with proper processing, the crowd performs just as well as medical experts in terms of the quality and efficacy of annotations, while being cheaper and more readily available. however, their results indicate that at least ten workers per sentence are needed to get the highest quality annotations, while we aim at acquiring high-quality results with fewer workers by assuring assignments of targets well-fit to the annotators. assignment of experts has been studied in the context of paper-reviewer assignment problem at academic conferences @xcite @xcite . these methods use previous publications of reviewers and those publications are used to build reviewers' profiles. methods for building profiles include not only use of words in publications, but also latent semantic indexing @xcite and latent dirichlet allocation @xcite . submitted papers are also encoded with the same methods for building reviewer profiles. then, the similarity between submitted papers and each reviewer's profile is calculated based on the encoded representations and assignment is done just with the calculated similarity or by casting assignment as an integer linear programming problem @xcite . our work is in the similar spirit as this line of work. the experimental results indicate that such an approach is also promising in selecting annotators for expert domain annotation tasks. to sum up the related work: current crowdsourcing-based annotation methods integrate several annotation results into one final annotation, utilize machine learning with several annotation results, or target experts where the most skillful annotators are automatically discovered. to the best of the authors' knowledge, the proposed task of automatic assignment of documents for annotation is the first of its kind. besides, this is the first research that evaluates the effectiveness of consideration of annotator's expertise in quality. ## experimental settings we evaluated the proposed annotator assignment method on a chemical name annotation task using the abstracts of chemistry papers. ## discussion and conclusion good quality corpora of specialized scientific texts could become important not only for specialists from various fields, but also for the promising ai subfield of automatic scientific discovery. this paper proposes a novel method for automatic annotation task assignment based on the expertise of the annotator estimated with scientific paper abstracts that the annotator has presented as relevant to their own research interests. the experimental annotation results of english and japanese chemistry-related paper abstracts showed that our method contributes to higher accuracy compared to the annotation by non-experts and use of predefined technical field categories. we could not cover all possible experimental settings (e.g. assignment using only implicit expertise) due to the time and budget restrictions. in our experimental design it is required that all assignments to be compared are calculated on the same set of annotators and then annotated together, which makes adding other experimental settings afterwards difficult. improvement in experimental design is needed in order to compare different factors more flexibly. an interesting observation is that the cause of annotation errors is not always the lack of annotator expertise. for example, we found that errors which result from poor understanding of annotation guidelines are not negligible. for the japanese task, in which we failed to find any significant improvement, as many as 20 of mention types might have made the annotation task too complicated for annotators that are domain experts but not experts in linguistic annotation. as shown in