Identifying C hinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations (2022)

Abstract: Recent works of opinion expression identification (OEI) rely heavily on the quality and scale of the manually-constructed training corpus, which could be extremely difficult to satisfy. Crowdsourcing is one practical solution for this problem, aiming to create a large-scale but quality-unguaranteed corpus. In this work, we investigate Chinese OEI with extremely-noisy crowdsourcing annotations, constructing a dataset at a very low cost. Following Zhang el al. (2021), we train the annotator-adapter model by regarding all annotations as gold-standard in terms of crowd annotators, and test the model by using a synthetic expert, which is a mixture of all annotators. As this annotator-mixture for testing is never modeled explicitly in the training phase, we propose to generate synthetic training samples by a pertinent mixup strategy to make the training and testing highly consistent. The simulation experiments on our constructed dataset show that crowdsourcing is highly promising for OEI, and our proposed annotator-mixup can further enhance the crowdsourcing modeling.

## introduction opinion mining is a fundamental topic in the natural language processing (nlp) community, which has received great attention for decades @xcite . opinion expression identification (oei) is a standard task of opinion mining, which aims to recognize the text spans that express particular opinions @xcite . figure 1 shows two examples. this task has been generally solved by supervised learning @xcite with the well-established corpus annotated by experts. almost all previous studies are based on english datasets such as mpqa @xcite . by carefully examining this task, we can find that the corpus annotation of opinion expressions is by no means an easy process. it is highly ambiguous across different persons. as shown in figure 1 , it is very controversial to define the boundaries of opinion expressions @xcite . actually, this problem is extremely serious for languages such as chinese, which is based on characters even with no explicit and clearly-defined word boundaries. thus, chinese-alike languages will inevitably involve more ambiguities. in order to obtain a high-quality corpus, we usually need to train the annotators with great efforts, making them acquainted with a specific finegrained guideline drafted by experts, and then start the data annotation strictly. finally, it is better with a further expert checking on borderline cases where the annotators disagree most to ensure the quality of the annotated corpus. apparently, the whole process is quite expensive. thus, crowdsourcing with no training (just a brief guideline) and no expert checking is more practical in real considerations @xcite . while on the other hand, the difficulty of the chinese oei task might lead to very low-quality annotations by crowdsourcing. in this work, we present the first study of chinese oei by using crowdsourcing. we manually construct an oei dataset by crowdsourcing, which is used for training. indeed, the dataset is cheap but with a great deal of noises according to our initial observation. we also collect the small-scale devel-opment and test corpus with expert annotations for evaluation. 1 our dataset is constructed over a set of chinese texts closely related to the covid-19 topic. following, we start our investigation by using a strong bert-bilstm-crf model, treating the oei task as a standard sequence labeling problem following the previous studies @xcite @xcite . our primary goal is to answer whether these extremely-noisy crowdsourcing annotations include potential value for the oei task. in order to make the best use of our crowdsourcing corpus, we follow @xcite to treat all crowd annotations as gold-standard in terms of different annotators. we introduce the annotatoradapter model, which employs the crowdsourcing learning approach of @xcite in oei for the first time. it jointly encodes both texts and annotators, then predicts the corresponding crowdsourcing annotations in the bert-bilstm-crf architecture. concretely, we train the annotatoradapter model by each individual annotator and the corresponding annotations, then test the model by using a pseudo expert annotator, which is a linear mixture of crowd annotators. considering that this expert is never modeled during the training, we further exploit a simple mixup @xcite strategy to simulate the expert decoding accurately. experimental results show that crowdsourcing is highly competitive, giving an overall f1 score of 53.86 even with a large-scale of noises, while the f1 score of expert corpus trained model is 57.08. we believe that this performance gap is totally acceptable for building oei application systems. in addition, our annotator-mixup strategy can further boost the performance of the annotator-adapter model, giving an f1 increase of 54.59 -53.86 = 0.73. we conduct several analyses to understand the oei with crowdsourcing and our suggested methods comprehensively. in summary, we make three majoring contributions as a whole in this work: ## dataset the outbreak of covid-19 brings strong demand for building robust chinese opinion mining systems, which are practically built in a supervised manner. a large-scale training corpus is the key to the system construction, while almost all existing related datasets are in english @xcite . hence, we manually construct a chinese oei dataset by crowdsourcing. we focus on opinion expressions with positive or negative polarities only. the construction consists of four steps: (1) text collection, (2) annotator recruitment, (3) crowd annotation, and (4) expert checking and correction. ## methodology the oei task aims to extract all polarized text spans that express certain opinions in a sentence. it can be naturally converted into a sequence labeling problem by using the bio schema, tagging each token by the boundary information of opinion expressions, where "b-x" and "i-x" (i.e., "x" can be either "pos" or "neg" denoting the polarity) indicate the start and other positions of a certain expression, and "o" denotes a token do not belong to any expression. in this work we adopt the crf-based system @xcite to the neural setting and enhance it with bilstm encoder as well as pre-trained bert representation. ## conclusion we presented the first work of chinese oei by crowdsourcing, which is also the first crowdsourcing work of oei. first, we constructed an extremely-noisy crowdsourcing corpus at a very low cost, and also built gold-standard dataset by experts for experimental evaluations. to verify the value of our low-cost and extremely-noisy corpus, we exploited the annotator-adapter model presented by @xcite to fully explore the crowdsourcing annotations, and further proposed an annotator-mixup strategy to enhance the model. experimental results show that the annotator-adapter can make the best use of our crowdsourcing corpus compared with several representative baselines, and the annotator-mixup strategy is also effective. our final performance can reach an f-score of 54.59% by exact matching. this number is actually highly competitive by referring to the model trained on expert annotations (57.08%), which indicates that crowdsourcing can be highly recommendable to set up a chinese oei system fast and cheap, although the collected corpus is extremely noisy.