UOR at S em E val-2021 Task 12: On Crowd Annotations; Learning with Disagreements to optimise crowd truth (2021)

Abstract: Crowdsourcing has been ubiquitously used for annotating enormous collections of data. However, the major obstacles to using crowd-sourced labels are noise and errors from non-expert annotations. In this work, two approaches dealing with the noise and errors in crowd-sourced labels are proposed. The first approach uses Sharpness-Aware Minimization (SAM), an optimization technique robust to noisy labels. The other approach leverages a neural network layer called softmax-Crowdlayer specifically designed to learn from crowd-sourced annotations. According to the results, the proposed approaches can improve the performance of the Wide Residual Network model and Multi-layer Perception model applied on crowd-sourced datasets in the image processing domain. It also has similar and comparable results with the majority voting technique when applied to the sequential data domain whereby the Bidirectional Encoder Representations from Transformers (BERT) is used as the base model in both instances.

## introduction in recent years, there has been some major advancement in the use of deep learning for solving artificial intelligence problems in different domains such as sentiment analysis, image classification, natural language inference, speech recognition object detection. they have also been used in many other numerous cases where human disagreements are encountered such as speech recognition, visual object recognition, object detection and machine translation @xcite . it is however, an essential requirement for deep learning models to utilise labelled data to undertake the representational learning of the underlying datasets. these labelled data are most at times not available and hence the need for humans to manually undertake the labelling of these data becomes a necessity. in recent years, crowd-sourcing has been used in the annotation of large collections of data and has proven to be an efficient and cost-effective means of obtaining labeled data as compared to expert labelling @xcite it has been utilised in the generation of image annotations to train computer vision systems @xcite , to provide the linguistic annotations used for natural language processing (nlp) tasks @xcite , and has also been used to collect the relevant judgments needed to optimize search engines @xcite . it is a well known fact that crowd-sourced labels are known to be associated with noise and errors as a result of the annotations being provided by annotators with uneven expertise and dedication which can result in the compromise of practical applications that uses such data @xcite . this paper therefore seeks to apply a novel approach to minimize and mitigate the noise and errors in crowd sourced labels. the aim is to investigate the use of a unified testing framework to learn from disagreements using crowd source labels collected from different annotators. ## related work crowdsourcing has proven to be an inexpensive and efficient way to collect large labels of data and has attracted much research interest from the machine learning community to address noise and unreliabilities associated with them. the proposal for using an expected maximization (em) algorithm to obtain density estimate rate of errors of patients providing conflicting responses to medical questions by @xcite , is one of the key pioneer contributions to this field. this work served as the catalyst for many other approaches used for the aggregation of labels from crowd annotators with different levels of expertise, such as the one proposed in @xcite , which further extends dawid and skene's model by also accounting for item difficulty in the context of image classification. similarly, @xcite proposed using dawid and skene's approach to extract a single quality score for each worker that low-quality workers to be pruned. the approach proposed in our paper contrast with this line of work, by allowing neural networks to be trained directly on the softmax output of the noisy labels of multiple annotators, thereby avoiding the need to resort to prior label aggregation schemes. @xcite also collated the opinions of many experts to establish ground truth and there has been a large body of research work using em approaches to annotate labels for datasets by many experts @xcite . @xcite also used the em approach of labelling datasets by experts through the use of gaussian process classifiers. @xcite also deployed the use of crowd layer with a cnn model to capture the biases of different annotators and correct them, our approach is the first to be built on the wide residual network (wideresnet) model @xcite and the bidirectional encoder representations from transformers (bert) model @xcite . our approach differs from the method used by @xcite because our technique initially finds the softmax of the output of the crowd responses before it is used for the modelling whereas the rodrigues and pereira (2018) approach works on the responses from the crowd directly. ## systems description these systems are proposed for image classification tasks and nlp tasks with sub-task-specific modifications and training schemes applied to each of the dataset. ## results and discussion the results were evaluated using two metrics known as f 1 score, referred to as hard evaluation and and cross entropy, referred to as soft evaluation. models with higher f 1 scores and lower cross entropy values are the desired outcomes expected from the models. as can be seen in refer to appendix a for the analysis of the class distribution of the datasets. ## conclusion this paper used a softmax-crowdlayer approach combined with a deep neural network to train noisy labels from multiple crowd annotators. wideres-net together with softmax-crowdlayer has been applied on cifar10-ic datasets, whilst mlp combined with softmax-crowdlayer has been used on the labelme-ic data and bert combined with softmax-crowdlayer has been used on gimpel-pos and pdis data respectively. future work will explore the effect of the distribution of the class annotation on the labeling accuracy and also investigate more efficient approaches of combining the bert model with the softmax-crowdlayer to further improve the results. it will also involve the application of the softmaxcrowdlayer on the humour dataset which was not included in this work due to time constraint posed as a result of the complicated data points of the humour dataset.