Understanding the Tradeoff between Cost and Quality of Expert Annotations for Keyphrase Extraction (2020)

Abstract: Generating expert ground truth annotations of documents can be a very expensive process. However, such annotations are essential for training domain-specific keyphrase extraction models, especially when utilizing data-intensive deep learning models in unique domains such as real-estate. Therefore, it is critical to optimize the manual annotation process to maximize the quality of the annotations while minimizing the cost of manual labor. To address this need, we explore multiple annotation strategies including self-review and peer-review as well as various methods of resolving annotator disagreements. We evaluate these annotation strategies with respect to their cost and on the task of learning keyphrase extraction models applied with an experimental dataset in the real-estate domain. The results demonstrate that different annotation strategies should be considered depending on specific metrics such as precision and recall.

## introduction automatic keyphrase extraction is an important technology on the crossroads of natural language processing and information access. domain-specific keyphrase extraction models are widely used in many real-world applications such as document characterization and clustering @xcite , domain specific knowledge organization @xcite , topic-based access to document collections @xcite , natural language question answering @xcite , personalized recommendation of external content @xcite , and many other tasks @xcite . one approach to such keyphrase extraction is to apply pre-trained or unsupervised models. however, such models might suffer from a lack of domain-specific knowledge. for example, while the terms "home" or "bedroom" alone can be called keyphrases in general, they carry very little information in the real-estate domain. therefore, training domain-specific keyphrase extraction models based on expert knowledge is vital in such applications. in addition, for some easier tasks such as content linking or content recommendation, automatic processing could support sufficient levels of quality. for more challenging tasks, such as personalization, the use of expert annotation in some form is essential. annotations of domain-specific documents can be performed via crowd-sourcing, online workers and/or expert annotators @xcite . this process can be done by each expert annotating a single document, doing self-review, multiple experts annotating the same document or doing peer-reviews. in the case of multi-annotator disagreement, a voting rule should be applied. in previous studies, a common practice to measure the quality of annotation labels is to compute inter-annotator agreement @xcite @xcite @xcite . more annotation/review steps involved in this process often result in a better agreement between annotators. however, it is still unclear whether the training set extracted from annotations with multiple annotation/review steps would increase the performance of keyphrase extraction models significantly. in addition, multiple annotation/review steps inevitably increase the cost of the annotation process as multiple experts have to tag the same set of documents. due to the cost of hiring domain experts, it is critical to select an annotation strategy that maximizes the quality while minimizing the cost. despite this substantial higher cost, to the best of our knowledge, there is no study analyzing the tradeoff of using annotations by multiple experts and the performance improvements of the keyphrase extraction model. to fill in this gap, this paper experimentally analyzes the tradeoff between the cost of annotation strategies and the impact these strategies have on the model performance. we measure the mean annotation time per document and compare the precision, recall and f1-score measures of classification and sequence labeling models over self-review, peer-review and different annotation aggregation methods. the experimental results demonstrate that different annotation strategies should be selected depending on whether the objective is to optimize for precision, recall or f1-score. the rest of the paper is organized as follows. section 2 reviews related work and section 3 discusses annotation and keyphrase extraction methodology. the experimental results are presented in 4 and the paper is concluded in 5. ## methodology 3.1 annotation procedure to evaluate the tradeoff between cost and quality of expert annotation labels, we analyze multiple methods and aggregation strategies (voting rules) for label generation. in this annotation process, three experts, who have at least six months experience of working in the real-estate domain, receive training and pass a test that focuses on the understanding of the task and the brat annotation interface 1 . they, then, develop guidelines (described in section 3.1.1) through multiple weekly annotation discussions. in each iteration, the experts independently label keyphrases in 10 listing descriptions, followed up with a discussion to resolve disagreement and updating the guidelines. next, the experts annotate a larger set of 50 listing descriptions and record the annotation time. the experts are not allowed to discuss their annotation/review before finishing all annotation tasks. all labels generated in the procedure are combined based on different strategies (described in section 3.1.2) and used as ground truth data for training classification and sequence labeling models. ## conclusions in this paper, we presented multiple annotation strategies including self-review and peer-review processes as well as various ways of resolving annotator disagreement for keyphrase annotation problems. we trained a classification model with an extensive list of features in the domain of real-estate. in addition, we applied a bi-lstm-crf architecture for a sequence labeling approach to extract keyphrases. we evaluated the two models' performances with eighteen different training datasets generated from the aforementioned strategies to see the tradeoff between the cost and quality of expert annotations. the results showed that different annotation strategies can be considered depending on a specific metric. we observed the consistent improvement for precision or recall when applying different voting rules for all the three review types. with respect to average f1-scores, we do not see an improvement of self-review and peer-review over the original annotations. the comparison between the two supervised models with the state-of-the-art unsupervised models has shown the importance of exploiting domain-specific knowledge and expert annotations to keyphrase extraction problems. however, this work is limited to one small dataset. it could be extended and evaluated on multiple datasets from different domains (e.g., e-commerce, education or medical) to examine the general applicability of our proposed annotation strategies. this work, to the best of our knowledge, is the first to understand the tradeoff between cost and quality of expert annotation for keyphrase extraction. there is still room to improve the models, for example by leveraging user search terms as a feature for logreg or pre-trained language models and transfer learning for bi-lstm-crf. our priority is to investigate whether the approach is valid for other nlp tasks such as relation extraction. we also plan to investigate how useful the generated code book for weak supervision modeling, how to translate the rules in the guidelines to labeling functions in weak supervision.