An Approach to the Frugal Use of Human Annotators to Scale up Auto-coding for Text Classification Tasks (2021)

Abstract: Human annotation for establishing the training data is often a very costly process in natural language processing (NLP) tasks, which has led to frugal NLP approaches becoming an important research topic. Many research teams struggle to complete projects with limited funding, labor, and computational resources. Driven by the Move-Step analytic framework theorized in the applied linguistics field, our study offers a rigorous approach to the frugal use of two human annotators to scale up auto-coding for text classification tasks. We applied the Linear Support Vector Machine algorithm to text classification of a job ad corpus. Our Cohenâs Kappa for inter-rater agreement and Area Under the Curve (AUC) values reached averages of 0.76 and 0.80, respectively. The calculated time consumption for our human training process was 36 days. The results indicated that even the strategic and frugal use of only two human annotators could enable the efficient training of classifiers with reasonably good performance. This study does not aim to provide generalizability of the results. Rather, we propose that the annotation strategies arising from this study be considered by our readers only if such strategies are fit for one’s specific research purposes.

## introduction in natural language processing (nlp), human annotation is an indispensable and decisive step. the human annotation process directly influences the quality of the training data in nlp tasks, and consequently, it influences the quality of machinegenerated results. in this regard, @xcite have revealed how significant the risk of reaching an incorrect conclusion could be if the quality of human annotation used for validation cannot be guaranteed. unfortunately, the science of annotation is progressing very slowly @xcite . in many nlp studies, methodological details concerning the human annotation process have not been fully disclosed @xcite . such a lack of disclosure may hinder readers' judgment of the soundness of human annotation procedures @xcite . it is time for nlp researchers to attach greater importance to the methodological rigor of human annotation in nlp tasks. where the funding and labor are limited, institutions or researchers might have to turn to the 'frugal' use of human annotators for text labelling tasks. for instance, @xcite acknowledged the limitation of not being able to afford high computational and labor costs in their machine learning (ml)-assisted analysis of tweeter commentary. @xcite also point out cost control that many engineering teams may need to deal with and emphasize the importance of minimizing labor cost and required training data to meet target results in nlp projects. therefore, well-planned investment of labor and training resources for nlp and ml tasks is a topic worth considerable scholarly attention. we need to investigate how to make the best use of limited labor and monetary resource to achieve the optimal machine-generated outcomes, while preserving methodological rigor. crowdsourcing is often put forward as a solution to the human coder resource problem. aside from the fact that crowds are often not experts, this kind of human annotation is allowed only in some national contexts, such as in the us (e.g., @xcite . this solution is not broadly applicable and has ethical implications with respect to researchers exploiting free or cheap labor. for instance, such option does not conform to the requirement for minimum hourly salaries under employment laws in national contexts such as australia @xcite . under circumstances of regulatory limitations and within ethical constraints, it becomes necessary to resort to the frugal use of human annotators to scale up data analyses. unlike human annotation tasks for ordinary image annotation (e.g., dog vs. cat recognition), many text annotations require expert knowledge because they are simply more demanding. for instance, the labelling of research skills in job ads involved human annotators who worked as researchers and educators at universities in @xcite . these researchers point out that it can be extremely time and money-consuming to hire multiple expert human annotators. in many cases, if annotation procedures were well-devised, the frugal option generated results that were as good as the more costly option @xcite . from the perspective of cost control, a better option would be to also involve non-expert annotators with well-designed annotation schemes to reach the optimal annotation outcomes @xcite . therefore, it is in the interest of textual-data scientists to investigate if there is a way to guarantee the quality of manual annotation with the frugal use of human coders for automatic textual data analyses at scale. as many social science disciplines (e.g., applied linguistics or sociology) have a record of excellent human annotation frameworks, it is worth considering if annotation frameworks in any of these fields could help us enhance the methodological soundness for human annotation process in nlp tasks. the research questions of this study are posed as follows: the primary outcomes of this study were as follows: ## results the inter-rater agreement measured by cohen's k reached an average of 0.76 (see section 3.1), meaning that most of our manually labeled categories can be used for making at least tentative conclusions. the results related to the total time investment in the human annotation process (see section 3.2) suggested that two human annotators, each working 5 hours a day, would need approximately 36 days to complete the task. section 3.3 is concerned with the performance of the two automatic classifiers trained with data labeled by our two annotators. although the two classifiers both reached an averaged auc of 0.80, a closer examination of fine-grained categories revealed potential room for further improvement to the human annotation schema. these findings posed the question of whether high-inter rater agreement is more important than the ml results' interpretability. moreover, strategic hybrid use of the two classifiers for optimization was introduced in section 3.3. ## discussion our study showed that even the frugal use of only two human annotators plus a limited amount of labeled data resulted in an averaged auc score of 0.80. nonetheless, the differentiation between the fine-grained and coarse-grained categories in our coding schema revealed even the averaged auc of 0.80 did not necessarily mean the quality of human annotation was as good 3 . the differentiation of fine and coarse granularities could enhance the interpretability of the results. in particular, such a differentiation provided a straightforward indication as to where the machine performed well or not and also where the problems lay in the human annotators' coding schema. our study had limitations. although we provided justifications for all the choices we made in our methods, there is room to refine our project's design (e.g., involving classification of other genres) when we have more resources. compared to most previous coding schemas where no differentiation of granularity levels was made, our approach could allow more to-the-point and efficient fixation of the human annotation for continuous improvement. our findings regarding the benefits of having two granularity levels echo the results in @xcite . our choice of making the differentiation between granularity levels counters the suggestion given by @xcite . they argue that coarser granularity would improve the accuracy of human annotation results. nonetheless, @xcite have mostly used examples of semantic recognition tasks such as verb-sense annotation to support their argument. our task of text classification is different from semantic recognition. therefore, it is worth further investigating whether 3 the point of constantly mentioning the coarse-grained categories in this paper is to emphasize how coarse granularity alone was unable to ensure the optimal performance for our specific annotation task. single granularity level has been pervasively used in many text classication tasks @xcite @xcite . nonetheless, recent studies @xcite @xcite suggest that single granularity cannot guarantee the optimal performance for certain tasks, which echo our findings here. in addition, we feel it necessary to keep the coarse granularity because the high-level categories are always useful when presenting complex results to the public it is reasonable to always opt for 'neutering' for all nlp tasks only for the sake of reaching a high inter-rater agreement regardless of the research purpose. our frugal use of one expert annotator and one non-expert annotators proved to cost moderate annotation time whilst generating reasonably good results. compared to the recruitment of multiple expert-annotators, our approach certainly was much less costly. the strategically hybrid use of automatic classifiers trained by our two annotators is perhaps comparable to a classifier trained by only expert annotators. however, such an assumption is subject to future investigations where appropriate measures are involved. future scholarly attempts could explore this topic of frugal hybrids of machines and human experts further to verify our assumption. in this regard, @xcite and @xcite echo our thoughts by arguing that a well-devised nonexpert annotator workshop could allow the labeling quality to be as good as when only expert annotators generate the labeling. @xcite expressed the concern that writing guidelines for even simple concepts for non-expert coders can be very prohibitive, but our approach of mixing both expert and non-expert coders is less likely to incur uncertainties and unexpected costs. to drive the progress of the science of annotation, scholars in the future might find it interesting to compare labeling results generated by pure experts, a mixture of experts non-experts, and crowdsourced workers for the same nlp project. ## conclusion in this study, we advocate a methodologically sound approach to the frugal use of two annotators to conduct human annotation tasks for nlp projects. our approach has multiple benefits. specifically, the time and resource consumption of our frugal approach were moderate compared to the more expensive choice of hiring multiple expert annotators. having multiple rounds of annotation activities and ongoing meetings makes it possible to make timely justification and adjustments for the annotation schema. moderate cost, timely communication of dubious labels, joint development of the annotation schema, and reasonably good ml outcomes are the features of our frugal but theoretically sound approach to human annotation. these features make the frugal use of minimally two hu-