Learning a C ost- E ffective A nnotation P olicy for Q uestion A nswering (2020)

Abstract: State-of-the-art question answering (QA) relies upon large amounts of training data for which labeling is time consuming and thus expensive. For this reason, customizing QA systems is challenging. As a remedy, we propose a novel framework for annotating QA datasets that entails learning a cost-effective annotation policy and a semi-supervised annotation scheme. The latter reduces the human effort: it leverages the underlying QA system to suggest potential candidate annotations. Human annotators then simply provide binary feedback on these candidates. Our system is designed such that past annotations continuously improve the future performance and thus overall annotation cost. To the best of our knowledge, this is the first paper to address the problem of annotating questions with minimal annotation cost. We compare our framework against traditional manual annotations in an extensive set of experiments. We find that our approach can reduce up to 21.1% of the annotation cost.

## introduction question answering (qa) based on textual content has attracted a great deal of attention in recent years @xcite @xcite . in order for state-of-the-art qa models to succeed in real applications (e.g., customer service), there is often a need for large amounts of training data. however, manually annotating such data can be extremely costly. for example, in many realistic scenarios, there exists a list of questions from real users (e.g., search logs, faqs, service-desk interactions). yet, annotating such questions is highly expensive @xcite @xcite : it requires the screening of a text corpus to find the relevant document(s) and subsequently screening the document(s) to identify the answering text span(s). motivated by the above scenarios, we study cost-effective annotation for question answering, whereby we aim to accurately foot_0 annotate a given set of user questions with as little cost as possible. generally speaking, there has been extensive research on how to reduce effort in the process of data labeling @xcite . for example, active learning for a variety of machine learning and nlp tasks @xcite aims to select a small, yet highly informative, subset of samples to be annotated. the selection of such samples is usually coupled with a particular model, and thus, the annotated samples may not necessarily help to improve a different model @xcite . in contrast, we aim to annotate all given samples at low cost and in a manner that can subsequently be used to develop any advanced model. this is particularly relevant in the current era, where a dataset often outlives a particular model @xcite . moreover, there has also been some research into learning from distant supervision @xcite or self-supervision @xcite . despite being economical, such approaches often produce inaccurate or noisy annotations. in this work, we seek to reduce annotation costs without compromising the resulting dataset quality. we propose a novel annotation framework which learns a cost-effective policy for choosing between different annotation schemes, namely the conventional manual annotation scheme (man) and a semi-supervised annotation scheme (sem). unlike the manual scheme, sem does not require humans to screen a text corpus or document(s) in order to retrieve annotations. instead, it leverages an initialized qa system, which can predict top-n candidate annotations for documents or answer spans and asks humans to provide binary feedback (e.g., cor- in such cases, the annotation cost has already been incurred and cannot be recouped. in order to produce an annotation, one must then draw upon the manual scheme (see fig. 1 ), in which case the policy would have been more effective if it had chosen the manual annotation scheme instead. therefore, how to choose the best annotation scheme for each question is the challenge we must address for this task. to tackle the above challenge, we propose a novel approach for learning a cost-effective policy. here the policy receives several candidates and decides on this basis which annotation scheme to invoke. we train the policy with a supervised objective and learn a cost-sensitive decision threshold. the inherent advantage of this method is that our policy immediately reacts to changing costs (without re-optimizing model parameters) and does not exceed the cost of conventional manual annotation. our policy is updated iteratively as more annotations are obtained. we compare our framework against conventional, manual annotations in an extensive set of experiments. we simulate the annotation of nat-uralquestions @xcite , as it consists of real user questions from search logs. models in our framework are initialized with an existing dataset @xcite and, as more annotations on naturalquestions be-come available, the framework is continuously updated. we study the sensitivity of our framework to varying cost ratios between sem and man. our framework outperforms traditional manual annotation, even under conservative cost estimates for sem, and in general reduces annotation costs in the range of 4.1% to 21.1%. all source code is publicly available from github.com/bernhard2202/qa-annotation. active learning: in the fields of machine learning and nlp, extensive research has been conducted on ways to reduce labeling effort (e.g., @xcite . for example, the objective of active learning is to select only a small subset that is highly informative (e.g., @xcite for annotation. to this end, researchers have developed various techniques based on, e.g., model uncertainty (cf. @xcite , expected model change @xcite , or functions learned directly from data (e.g., @xcite . however, the success of active learning is often coupled with a particular model and domain @xcite . for instance, a dataset actively acquired with the help of an svm model might underperform when used to develop an lstm model. these problems become even more salient when complex black-box models are used in nlp tasks (cf. @xcite . to summarize, active learning reduces annotation costs by deciding which samples should be annotated. in our approach, we aim to annotate all samples and study how we should annotate them in order to reduce costs. thus, the two approaches are orthogonal and can be combined. learning from weak supervision and user feedback: another approach to reducing annotation costs is changing full supervision to some form of weak (but potentially noisier) supervision. this has been adopted for various tasks such as machine translation @xcite @xcite , semantic parsing @xcite @xcite , or interactive systems that learn from user interactions @xcite @xcite . for instance, @xcite used users to flag incorrect sql queries. in contrast, similar approaches for text-based question answering are scarce. @xcite used noisy distant supervision to annotate the answer span and document for given trivia questions and their answers. @xcite designed a qa system that continuously learns from noisy user feedback after deployment. in contrast to these works, this paper studies the problem of reducing labeling cost while maintaining accurate annotations. quality estimation and answer triggering: in a broader sense, this work is related to the literature on translation quality estimation (e.g., @xcite . the goal in such works is to estimate (and possibly improve) the quality of translated text. similarly, in question answering researchers use means of quality estimation for answer triggering @xcite . here, qa systems are given the additional option to abstain from answering a question when the best prediction is believed to be wrong. in our work, we estimate the quality of a set of suggested label candidates and, on the basis of these estimates we decide which annotation scheme to invoke. ## proposed annotation framework we study the problem of reducing the overall cost for annotating every given question [q 1 , . . . , q m ]. specifically, our objective is to obtain the corresponding question-document-answer triples q i , d i , s i . in this paper, the natural language question q i is given, while we want to obtain the following annotations: the document from a text corpus d @xmath0. ## experimental setup in this section, we introduce our experimental setup and implementation details. ## experimental results we group our experiments into three parts. first, we focus only on annotating the answer span for given question-document pairs, as this is the more challenging task. 4 second, we carry out a sensitivity analysis in order to demonstrate how our framework adapts to different costs of sem annotations and to show that we never exceed the cost of traditional annotation. third, we evaluate our framework based on the annotation of a full dataset, including both answer span and document annotations, in order to quantify savings in practice by using our framework. ## conclusion we presented a novel annotation framework for question answering based on textual content, which learns a cost-effective policy to combine a manual annotation scheme with a semi-supervised annotation scheme. our framework annotates all given questions accurately while limiting costs as much as possible. we show that our framework never incurs higher costs than traditional manual annotation. on the contrary, it achieves substantial savings. for example, it reduces the overall costs by about 4.1% when sem annotations cost about half of man annotations. when that ratio is lowered to one fourth, our framework can reduce the total costs by up top 21.1%. we think that our framework could contribute to more accessible annotation of datasets in the future and possibly even be extended to other fields and applications in natural language processing.