HW - TSC at S em E val-2022 Task 3: A Unified Approach Fine-tuned on Multilingual Pretrained Model for P re TENS (2022)

Abstract: In the paper, we describe a unified system for task 3 of SemEval-2022. The task aims to recognize the semantic structures of sentences by providing two nominal arguments and to evaluate the degree of taxonomic relations. We utilise the strategy that adding language prefix tag in the training set, which is effective for the model. We split the training set to avoid the translation information to be learnt by the model. For the task, we propose a unified model fine-tuned on the multilingual pretrained model, XLM-RoBERTa. The model performs well in subtask 1 (the binary classification subtask). In order to verify whether our model could also perform better in subtask 2 (the regression subtask), the ranking score is transformed into classification labels by an up-sampling strategy. With the ensemble strategy, the performance of our model can be also improved. As a result, the model obtained the second place for subtask 1 and subtask 2 in the competition evaluation.

## introduction as we all know, the proposal of bert @xcite , which is based on masked language modeling, is a huge milestone in the history of natural language understanding @xcite . compared with various language representation models, bert successfully pushes the glue score at 7.7 points absolute improvement @xcite . soon, different types of language models such as xlnet @xcite , roberta @xcite ,mbert @xcite and xlm @xcite are also proposed. compared with some strong monolingual models introduced above, xlm-roberta is more competitive on the glue and xnli benchmarks @xcite . there are lots of pretrained models proposed in recent years, which are capable of learning the implicit knowledge. some works have proved that the neural language model has learnt the implicit linguistic knowledge and this knowledge can significantly affect the predictions through finetuning @xcite . it has been believed that the pretrained models trained on wikipedia and other datasets already have some implicit knowledge. implicit knowledge could be classified into two categories: the connection between two objects, and the implicit logic and syntax behind the sentence. it means, we need to build a model which is capable of verifying the rationality and reliability of sentences, testing whether the latent knowledge can be expressed explicitly. presupposed taxonomy is a kind of concept in computational linguistics. two arguments could have several different taxonomy relationships. for example, the sentence "i like piano, but not the instrument" is a classic pattern which contains two arguments. in the transition sentence, "piano" and "instrument" have a taxonomic relation, so the conclusion could be drawn that the sentence is implausible and unacceptable. similarly, there are a set of sentences that could have such contradiction in the competition. the goal of the semeval-2022 task 3 @xcite is to recognize the presupposed taxonomy relation between two nouns, which could be a complex linguistic problem. in order to obtain the skill, the model needs to understand the implicit meaning of the sentences as well as whether the presupposed taxonomy relation exists in one pair of entities. id sentence labels 5155 i like ham, but not fish. 1 2560 i like restaurants, and clerks too. 1 3711 i like jewlry more than skirts. 1 4481 i like scientists more than geneticists. 0 3104 i do not like seafood, i prefer salmon. 0 along the lines of ideas above, there are several representative pretained-models in the competition. to be specific, we tried commonly used pretrained models including bert, roberta-large, and multilingual language models like xlmr. at the same time, we suspect that the former model can be generalized to subtask 2. the final ranking result supports our hypothesis strongly. subtask 1 is a binary classification task aiming at predicting the acceptability of sentences (a(1) vs ua(0)). there are two parameters that have the presupposed taxonomy relation, followed by the label "0"(the semantic relation is not acceptable) or "1" (the semantic relation is acceptable). note that label "0" stands for the contradictory sentences, and label "1" stands for the plausible sentence. some samples are shown in in this multilingual task, three languages (french, italian and english) are included. on the contrary, subtask 2 is a regression task aiming at predicting the degree of acceptance in a seven likert-scale. the only difference between the dataset of subtask 1 and subtask 2 is the label. in subtask 2, the label(score) is a float number between 1 and 7, which indicates the acceptable degree of the sentence. some english data samples are shown in in addition, based on a suitable single model and adaptive finetuning models which have learnt enough implicit information, it becomes possible to express the explicit knowledge(taxonomies) for a model. the baseline model provided by competition reaches accuracy at 0.8, which is an amazing result. we also find the training set is extremely unbalanced in the proportion of positive and negative samples for some patterns, as we can see in ## results and analysis except for the experiments introduced above, there were still a lot of directions we tested, but not all of them were useful. then, we classified it with a binary classifier (softmax). but there were no improvements in the performance.