Temporal-aware Language Representation Learning From Crowdsourced Labels (2021)

Abstract: Learning effective language representations from crowdsourced labels is crucial for many real-world machine learning tasks. A challenging aspect of this problem is that the quality of crowdsourced labels suffer high intra- and inter-observer variability. Since the high-capacity deep neural networks can easily memorize all disagreements among crowdsourced labels, directly applying existing supervised language representation learning algorithms may yield suboptimal solutions. In this paper, we propose TACMA , a temporal-aware language representation learning heuristic for crowdsourced labels with multiple annotators. The proposed approach (1) explicitly models the intra-observer variability with attention mechanism; (2) computes and aggregates per-sample confidence scores from multiple workers to address the inter-observer disagreements. The proposed heuristic is extremely easy to implement in around 5 lines of code. The proposed heuristic is evaluated on four synthetic and four real-world data sets. The results show that our approach outperforms a wide range of state-of-the-art baselines in terms of prediction accuracy and AUC. To encourage the reproducible results, we make our code publicly available at https://github.com/CrowdsourcingMining/TACMA .

## introduction crowdsourcing offers the ability to utilize the power of human computation to generate data annotations that are needed to train various ai systems. for many practical supervised learning applications, it may be infeasible (or very expensive) to obtain objective and reliable labels due to many reasons such as varying skill-levels and biases of crowdsourced workers. instead, to improve the quality of labels, we can collect subjec- * corresponding author: zitao liu. meanwhile, with the recent advances of deep neural networks (dnns), supervised representation learning (srl) has led to rapid improvements in the ability of learning intrinsic nonlinear embeddings using dnns that preserves the distance between similar examples close and dissimilar examples far on the embedding space. in spite of the significant progress for srl applications such as face recognition @xcite , image retrieval @xcite , directly applying existing deep language representation learning approaches on crowdsourced labels may yield poor generalization performance @xcite . because of the high capacity, dnns could entirely memorize the inconsistency within crowdsourced labels sooner or later during the modeling training process. besides, this phenomenon does not change with the choice of training optimizations or network architectures @xcite . a large spectrum of approaches have been successfully developed in either estimating true labels from crowdsourced labels, a.k.a., truth inference or label aggregation @xcite , learning via adversarial data generation @xcite , or learning language representations discriminatively from large-scale consistent labeled data with complicated neural architectures @xcite . however, learning effective neural embeddings directly from crowdsourced labels of real-world data poses numerous challenges. first, crowdsourced workers conduct labeling tasks sequentially, i.e., they label samples one after another. such sequential labeling behavior is a process of learning, and the expertise of the workers is not stable but gradually changing even without feedback @xcite . according to miller's law @xcite , humans retain what they just learned in their shortterm working memory with a limited span of 7 ± 2. temporal factors such as fatigue @xcite and intrinsic motivation @xcite implicitly influence the crowdsourcing quality, which are different from existing well-studied factors, such as the quality of crowdsourced workers, the difficulty of data samples, the price of annotation tasks, etc. in the following, such unconscious temporal behaviors are referred to as "temporal labeling effects". how to model such sample-level temporal information for each individual worker undoubtedly poses a hard modeling problem. second, a large number of real-world crowdsourced data sets have a substantial amount of disagreement among labels and a relatively small sample size. the majority of existing srl approaches are discriminatively trained on large-scale consistent labeled data to learn their complicated neural architectures, which may easily overfit the inconsistent crowdsourced data. in this paper we study and develop solutions that are applicable and can learn effective neural language representations from crowdsourced labels in an end-to-end manner. our work focuses on the refinements of a popular deep language representation learning paradigm: the deep metric learning (dml) @xcite @xcite . we aim to develop an algorithm to automatically learn a nonlinear language representation of the crowdsourced data from multiple workers using dnns. briefly, the dml is a classical and widely used approach for language representation learning that preserves the distance between similar examples close and dissimilar examples far on the embedding space. the majority of existing dml techniques restricted to just noise-free labels appropriately. however, learning effective representation from highly inconsistent crowdsourced data sets from multiple workers gives rise to numerous important questions: (1) since in practice, annotation performance is affected and varied over time @xcite , how do we capture such temporal labeling effects in the dml learning framework? (2) while in some cases the problem may be alleviated by pre-processing methods, such as filtering @xcite , label correction @xcite , truth inference @xcite , etc., the number of remained instances is often significantly reduced or such preprocessing errors for many problems will be propagated to the downstream representation learning tasks. how to capture the label uncertainties from multiple workers and at the same time prevent the overfitting problem in an end-to-end framework? in this work we address the above issues by presenting a temporal-aware language representation learning heuristic for crowdsourced labels with multiple annotators (tacma), that • utilizes the attention mechanism to capture the temporal influence among sequential labeling tasks according to each worker's short-term working memory. ## experiments experiments are conducted on both real-world and synthetic data sets. the internal cross validation approach is used to select hyper parameters when optimizing models' predictive performances. means as well as standard deviations of both accuracy and auc scores are reported, to comprehensively evaluate the performance of our proposed method, i.e. tacma. ## conclusion we presented tacma, an end-to-end framework for language representation learning from crowdsourced labels. comparing with traditional srl approaches, the advantages of our framework are: (1) it is able to consider temporal labeling effects within sequences of sample-level labeling tasks for each worker; (2) it automatically computes and aggregates sample-level confidence scores from multiple workers which makes the training process more effective. experimental results on both synthetic and real-world data sets demonstrates that our approach outperforms other state-of-the-art baselines in terms of accuracy and auc scores.