L ing J ing at S em E val-2022 Task 3: Applying D e BERT a to Lexical-level Presupposed Relation Taxonomy with Knowledge Transfer (2022)

Abstract: This paper presents the results and main findings of our system on SemEval-2022 Task 3 Presupposed Taxonomies: Evaluating Neural Network Semantics (PreTENS). This task aims at semantic competence with specific attention on the evaluation of language models, which is a task with respect to the recognition of appropriate taxonomic relations between two nominal arguments. Two sub-tasks including binary classification and regression are designed for the evaluation. For the classification sub-task, we adopt the DeBERTa-v3 pre-trained model for fine-tuning datasets of different languages. Due to the small size of the training datasets of the regression sub-task, we transfer the knowledge of classification model (i.e., model parameters) to the regression task. The experimental results show that the proposed method achieves the best results on both sub-tasks. Meanwhile, we also report negative results of multiple training strategies for further discussion. All the experimental codes are open-sourced at https://github.com/WENGSYX/Semeval .

## introduction in order to dive into the capability of the current language models @xcite , we take part in and apply the pre-training language model on the semeval-2022 tasks 3: the presupposed taxonomies: evaluating neural network semantics (pretens) @xcite . to evaluate the model performance comprehensively, two sub-tasks are designed in this pretens task: sub-task 1) binary classification task 2 , which consists in predicting the acceptability label assigned to each sentence of the test set. for example, "i like trees, and in particular birches" is acceptable while " i like oaks, and in particular trees" is unacceptable, so they are labeled 1 and 0, respectively. sub-task 2) regression sub-task foot_0 , which consists in predicting the average score assigned by human annotators on a seven-point likert scale @xcite with respect to the subset of data evaluated via crowdsourcing. for example,"i like governors, an interesting type of politician" is more acceptable than "i like politicians, an interesting type of farmer", so the former will also have a higher score (6.16) than the latter (1.42). it is noted that both sub-tasks comprise datasets in 3 languages: english, italian, french, where french and italian are slightly adapted translations of the english dataset. for each sub-task, every sample is formed as the arguments a and b, e.g., comparatives (i like a more than b), exemplifications (i like a, and in particular b), generalizations (i like a, and b in general), and others, where the argument nouns are taken from various semantic categories. the most similar tasks are the natural language inference @xcite @xcite and taxonomy expansion & enrichment @xcite @xcite , where the former requires the model to differentiate the relationship between a premise sentence and a hypothetical sentence, while the latter shall identify the relationship between different concepts. there are many powerful language models for accomplishing these tasks @xcite , and wellformed semantic representations can be obtained for the downstream tasks. however, it is a challenge for the model to decide the acceptance of the given sentence, as the semantic meaning is hard to be distinguished. moreover, the performance of downstream tasks when fine-tuning is limited by the size of the training dataset @xcite . to solve the above problems, we propose a method that applies deberta to lexical-level presupposed relation taxonomy with knowledge transfer. specifically, the powerful deberta-v3 pretrained model @xcite is fine-tuned with datasets of different languages in the classification task. for the regression task, due to the limited size of the training datasets, we fine-tune the trained model of the classification task in the regression task. as a result, the proposed method achieves a global top score of 94.173 in sub-task 1 and a global top score of 0.802 in sub-task 2. our method wins on two sub-tasks. in addition, we present the negative results of multiple training strategies when fine-tuning and provide further discussions. ## main method in this section, we will elaborate on the main methods for the two sub-tasks of the pretens task. the training strategies are included at the end of this section. ## results and discussions in this section, we studied the experimental results of the two sub-tasks and the impact of different strategies on the results, and further discussed the comparison with other participants to prove the effectiveness of the method. finally, studies analyze the deviation of language model and the future research direction.