Mao-Zedong at S em E val-2023 Task 4: Label Represention Multi-Head Attention Model with Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification (2023)

Abstract: This is our system description paper for ValueEval task. The title is:Mao-Zedong At SemEval-2023 Task 4: Label Represention Multi-Head Attention Model With Contrastive Learning-Enhanced Nearest Neighbor Mechanism For Multi-Label Text Classification,and the author is Che Zhang and Pingan Liu and ZhenyangXiao and HaojunFei. In this paper, we propose a model that combinesthe label-specific attention network with the contrastive learning-enhanced nearest neighbor mechanism.

## introduction the identification and analysis of human values in texts has been an important area of research. with the development of computational linguistics, this research has gained widespread attention because of its potential impact on areas such as sentiment analysis, social science. one of the challenges in this area is to accurately categorize all the human value. several notable research achievements have been made in the categorization of human values. one of the approach is that classifies human values into 54 categories across four different levels @xcite . se-meval2023 task4 uses the classification method in this paper, where an argument is given to identify whether a value is included in the instrument, and the f1 scores of the results at the level2 level are used for the total ranking. there are 20 cate-gories of human values in level 2, and a argument could belong to multiple value categories or not to any one value category. this is a typical multilabel text classification (mltc) problem which has been applied in many scenarios such as news emotion analysis @xcite and web page tagging @xcite . in this paper, we propose a model that combines the label-specific attention network with the contrastive learning-enhanced nearest neighbor mechanism @xcite . the multi-headed attention mechanism allows our model to overcome the shortcomings of traditional attention mechanism models and to be able to focus on different parts of a document, resulting in more accurate labeled attention results. and the nearest neighbor mechanism enables our model to not waste the rich knowledge that can be directly obtained from the existing training instances and helps enhance the interpretability and robustness of the model. ## system overview in this section, we will present our model, which consists of two main parts. the first part is a multiheaded attention mechanism based on a specific label representation, while the second part is a nearest neighbor mechanism enhanced using contrast learning. the mltc problem can be described as follows: assuming a set of data @xmath0, y i )} n @xmath1, n labeled documents, where x i represents the text and y @xmath2, 1} l represents the label of x i , and l represents the total number of labels. each document x i consists of a series of words. our goal is to learn a classifier to establish a mapping from x i to y i , so that when a new document x is presented, its label vector y can be correctly predicted.as pretrained language models (plms) show remarkable performance in extracting natural language representations, we use plms as base encoder to get document and label feature. a input sample can be expressed as x @xmath3, w 2 , w 3 . . . w n-1 , w n }, w @xmath4. after calculated by plms , the input matrix of the whole sentence is obtained as @xmath5, where d is the hidden dimension of plms. ## experimental setup in the dataset, many pronouns in the premise were used such as "this research", "it", "this way" etc, and these pronouns refer to objects contained in the conclusion. whereas our model tries to establish the attention scores of different semantic components of a document for a specific label, it is clear that the presence of these words with unclear denotations affects the attention results. in addition, stance toward conclusions also influence value judgments. therefore, we use a simple strategy: combining the three parts of conclusion, preference, and stance into a sentence that conforms to natural language conventions. specifically, the data were preprocessed uniformly and simply, and the input structure was: "i agree (disagree) that" + conclusion content + ", because" + premise content. we use the roberta model @xcite as the base pre-trained model to obtain word representation. experimenting with our architecture on the base model. and we use the k-fold cross-validation method. we merge the training and validation sets and then randomly divide them into six copies. we perform the training process six times, each time using 5/6 of the data as the training set and 1/6 of the data as the validation. during the training process, the best-trained model for each fold is saved, and the average output probability of all models is taken as the final prediction score. ## results on the leaderboard of the tira, our method achieved an macro-f1 score of 0.53 and ranks fourth, while the baseline which use a bert model @xcite achieved 0.42, with the best result for the whole competition being 0.56. in addition, our model also achieved an f1-score of 0.32 on each of the two test sets, nahj al-balagha and new york times. this effect is relatively high among all participating teams, which fully demonstrates the robustness and stability of our approach. 1 to illustrate the effectiveness of our architecture, we conducted ablation experiments. the ablation experiments evaluate the performance effect of the model directly on the validation set merged with the dataset from zhihu. in the ablation experiments, we did not use the strategy of k-fold cross-validation. the results of the ablation experiment are shown in ## conclusion we propose a multi-label text classification model using a label-specific multi-headed attention mechanism. compared to previous models of attention mechanisms, the use of multi-headed attention enables specific labels to focus on different semantic components of the document more effectively. besides, we use the knn mechanism to exploit the instance information in the training set. we then perform ablation experiments on our architecture to analyze the role of each part and demonstrate the superiority of using a multi-headed attention mechanism.