B ayesian Methods for Semi-supervised Text Annotation (2020)

Abstract: Human annotations are an important source of information in the development of natural language understanding approaches. As under the pressure of productivity annotators can assign different labels to a given text, the quality of produced annotations frequently varies. This is especially the case if decisions are difficult, with high cognitive load, requires awareness of broader context, or careful consideration of background knowledge. To alleviate the problem, we propose two semi-supervised methods to guide the annotation process: a Bayesian deep learning model and a Bayesian ensemble method. Using a Bayesian deep learning method, we can discover annotations that cannot be trusted and might require reannotation. A recently proposed Bayesian ensemble method helps us to combine the annotators’ labels with predictions of trained models. According to the results obtained from three hate speech detection experiments, the proposed Bayesian methods can improve the annotations and prediction performance of BERT models.

## introduction recent successful applications of artificial intelligence in various fields, including natural language processing, are often due to long hours of human annotation when preparing datasets for machine learning. the annotation process transfers human knowledge to machine learning models but it is often done under time pressure and with inadequate instructions or with insufficiently trained annotators.aiming to make the annotation process easier, we study the possibility of designing a data labeling process which requires less human supervision. in practice, a fairly standard procedure in the annotation quality control is to recheck the labels that are wrongly classified by using several prediction models. as an alternative, bayesian inference produces a distribution of possible decisions and can improve the selection of instances requiring reannotation @xcite . most neural networks do not support the assessment of predictive uncertainty. the bayesian inference framework can be helpful, however, most techniques do not scale well in neural networks with high dimensional parameter space @xcite . various methods were proposed to overcome this problem @xcite , one of the most efficient being monte carlo dropout (mcd) @xcite . its idea is to use the dropout mechanism in neural networks as a regularization technique @xcite and interpret it as a bayesian optimization approach that samples from the approximate posterior distribution. a common problem in text annotations is that annotators are not always sure about correct labels due to uncertainty in the text @xcite . on difficult texts, annotators frequently give ambiguous labels and their annotations can be biased. instead of asking annotators to label the raw text, it would be easier for them if they were proposed answers accompanied by probabilistic scores from an ensemble of predictive models. ensemble methods produce robust models that frequently provide significantly better predictions than individual models. the key strength of ensembles is that they can overcome errors and shortcomings of individual ensemble members. however, diversity in combining different predictions and reliability of individual predictions need to be better understood and evaluated @xcite . a recently published ensemble method multivariate normal mixture conditional likelihood model (mm) @xcite tries to understand the predictors on the distributional level and use bayesian inference to combine them. in this work, we evaluate mm's performance when combining predictive models on the hate speech detection task. we show that our methodology can serve as a helpful tool in the data annotation process. recently, the most successful approach in text classification is to use transformer neural networks @xcite , pretrained on large monolingual corpora, and then fine-tune them for a specific task, such as text classification. for example, bert (bidirectional encoder representations from transformers) @xcite uses masked language modeling and order of sentences prediction tasks to build a general language understanding model. during the fine-tuning for a specific downstream task, additional layers are added to the bert model, and the model is trained on the data of interest to capture the specific knowledge required to perform the task. the main aims of the paper is to propose methods that can save time and resources during the text annotation process and improve prediction performance. as a test domain we use hate speech detection in tweets, news comments and facebook comments. we investigate two performance improving techniques which can be summarized as our main contributions as follows. the paper consists of five further sections. in section 2, we present related works on prediction uncertainty and hate speech detection. in section 3, we propose the methodology for uncertainty assessment of deep neural networks using attention layers and mcd. in section 4, we describe the tested datasets and evaluation scenarios. the obtained results are presented in section 5, followed by conclusions and ideas for further work in section 6. ## related work in this section, we introduce related work split into four topics. first, we present the work on semisupervised learning that can be used in text annotation, followed by the related research on bayesian learning for text classification. in the third subsection, we describe probabilistic ensemble methods and in the fourth, we outline the related work on hate speech detection. ## methods we describe two approaches to the assessment of prediction reliability, bayesian attention networks and bayesian probabilistic ensembles. ## experimental setting we first introduce the three phases of our experiments, followed by the used datasets and implementation details. the experimental setting consists of three phases: ## results in this section, we present three groups of results: removing uncertain instances from the training set, creating a cleaner training set, and improving annotations using he bayesian ensembles. ## conclusions and further work a large amount of currently available textual data allows and requires modeling with machine learning methods. to apply the supervised methods, the text data has to be annotated, and effective learning requires accurate annotations, which may be expensive for organizers and difficult for human annotators. for this reason, the annotation process is often coupled with semi-supervised machine learning and classification reliability estimation. we presented several machine learning approaches, based on bayesian inference, that can improve the data annotation process. first, multiple predictions obtained with mcd bert can identify instances with questionable labeling. second, removing training instances with unreliable labels can improve the quality of the training set, making it more homogeneous and cleaner, thereby improving the predictive performance of bert models. third, probabilistic ensemble combinations can help annotators to better label the data by providing more accurate and better calibrated prediction probabilities. in conclusion, bayesian methods can improve the annotation process and shall be further investigated and improved for this task. in further work, we will focus on improving our method on how to remove uncertain instances. we will construct and test a workflow for semi-supervised text annotation in a real-world setting. testing different dropout levels in the bert model may provide a better understanding of its uncertainty and calibration.