Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning (2023)

Abstract: Fine-grained entity typing (FET) is an essential task in natural language processing that aims to assign semantic types to entities in text. However, FET poses a major challenge known as the noise labeling problem, whereby current methods rely on estimating noise distribution to identify noisy labels but are confused by diverse noise distribution deviation. To address this limitation, we introduce Co-Prediction Prompt Tuning for noise correction in FET, which leverages multiple prediction results to identify and correct noisy labels. Specifically, we integrate prediction results to recall labeled labels and utilize a differentiated margin to identify inaccurate labels. Moreover, we design an optimization objective concerning divergent co-predictions during fine-tuning, ensuring that the model captures sufficient information and maintains robustness in noise identification. Experimental results on three widely-used FET datasets demonstrate that our noise correction approach significantly enhances the quality of various types of training samples, including those annotated using distant supervision, ChatGPT, and crowdsourcing.

## introduction fine-grained entity typing (fet) is a fundamental task in the field of natural language processing (nlp). it involves assigning specific types to entity mentions based on the contextual information surrounding them. the results of fet can be leveraged to enhance various downstream nlp tasks, including entity linking @xcite , relation extraction @xcite , question answering @xcite and other tasks @xcite . fet poses a formidable challenge that entails not only the classification of entities across diverse domains but also the distinction of entities that * yongxiu xu is the corresponding author may exhibit superficial resemblances but possess distinct underlying meanings. for example, the term "apple" could pertain to a technology corporation or a fruit. achieving high performance in fet typically requires a considerable amount of annotated data, a resource that is both costly and time-consuming to acquire. to tackle this issue, researchers have explored various approaches for automatically labeling training samples. one particularly popular method is distant supervision @xcite , which assumes that any text mentions an entity in existing knowledge bases (e.g., freebase @xcite ) is related to the corresponding entity types. however, this technique can be susceptible to noise and ambiguity (refer to figure 1a ), since it fails to consider the contextual information surrounding the entities. meanwhile, the large language models (llms), such as chatgpt foot_0 , has drawn significant interest among researchers in exploring their applications in text annotation tasks. recent studies @xcite indicate that zeroshot chatgpt classifications outperform crowdworkers. however, despite their remarkable suc-cess in other domains, the weakly labels chatgpt generated for fet still pose challenges. as shown in figure 1b , it can generate accurate weakly labels, but its coverage of fine-grained types may be limited. this difficulty may arise from the challenge of designing a prompt that can accurately handle fine-grained entity types across multiple domains simultaneously. however, this incomplete recognition also results in noisy labels, as it mislabels the correct type as a negative sample. training deep models on data with noisy labels can result in the learning of incorrect patterns and subsequent incorrect predictions. as a result, several approaches have been proposed to tackle the issue of noisy labels in fet, including designing robust loss functions @xcite , estimating noise transition matrices @xcite , and correcting noisy labels @xcite @xcite . among these approaches, correcting noisy labels is often considered the most desirable, as it leads to more accurate and informative labels. however, the methods for noise correction often rely on estimating the distribution of noise, which can be challenging due to the diverse deviations arising from different labeling methods. for instance, distant supervision tends to generate inaccurate labels, while labels generated by chatgpt are accurate but incomplete. in this paper, we propose a novel noise correction method for fet using the co-prediction prompt-tuning technique, which leverages multiple prediction results to identify and correct noisy labels. specifically, we first fine-tunes a pre-trained masked language model (plm) on data with noisy labels using a co-prediction prompt. this prompt contains two mask tokens, which focus on different predictive capabilities and produce multiple predictions for each entity mention. note that the presence of noisy labels may make it difficult for the two masks to arrive at a consensus on the outputs. in this scenario, one mask may start fitting the noise before another, leading to divergent copredictions. hence, we design an optimization objective concerning divergent co-predictions during fine-tuning to maintain the fitting difference of masks on the noise labels and robustness to noise identification. afterward, we integrate the multiple prediction results to recall unlabeled labels and use a differentiated margin to identify inaccurate labels. we conduct experiments on three public fet datasets: ontonotes and wiki have distantly annotated training data; ultra-fine has crowdsourced training data. the experimental results show that the performance of a baseline model is significantly improves after training on the corrected data, indicating the effectiveness of our noise correction method. additionally, we utilized chatgpt to relabel a subset of training data from ontonotes and wiki and then optimized this data using our method. the experimental results demonstrate that our method can further improve the quality of the data annotated by chatgpt. ## methodology in this section, we introduce the problem formulation and describe our noise correction method. as shown in figure 2 , our method mainly consists of two main steps: 1) applying a co-prediction prompt to fine-tune plms with weakly labeled corpora; 2) correcting noisy labels by analyzing the co-prediction results of this model. ## conclusion this paper presents a novel noise correction method for fet through the utilization of co-prediction prompt tuning. our approach is not only straightforward but also highly effective, leveraging a pretrained language model that has been fine-tuned with a co-prediction prompt. this fine-tuned model is capable of identifying and rectifying noisy labels. to evaluate the performance of our method, we con-ducted experiments on three publicly available fet datasets, each containing different types of training samples, including those annotated through distant supervision, chatgpt, and crowdsourcing. the results of our experiments demonstrate the versatility of our noise correction method, as it significantly enhances the quality of the training data in all three scenarios. consequently, it leads to a substantial improvement in the overall performance of the fet model.