SPDB Innovation Lab at S em E val-2022 Task 3: Recognize Appropriate Taxonomic Relations Between Two Nominal Arguments with ERNIE - M Model (2022)

Abstract: Synonym and antonym practice are the most common practices in our early childhood. It correlated our known words to a better place deep in our intuition. At the beginning of life for a machine, we would like to treat the machine as a baby and built a similar training for it as well to present a qualified performance. In this paper, we present an ensemble model for sentence logistics classification, which outperforms the state-of-art methods. Our approach essentially builds on two models including ERNIE-M and DeBERTaV3. With cross validation and random seeds tuning, we select the top performance models for the last soft ensemble and make them vote for the final answer, achieving the top 6 performance.

## introduction synonym, antonym and their relations from unstructured text are fundamental problems in information classification field. these problems can be decomposed into three subtasks: word extraction using regrex, relation extraction @xcite , @xcite , and classifying the logistics between them. however, an end-to-end model, i.e. ernie-m model @xcite , is proposed to solve the three tasks. presupposed taxonomies -evaluating neuralnetwork semantics (pretens) @xcite is a task to predict the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. in this paper, we first present a simple approach with the ernie-m model to solve the task. although the ernie-m model performs unexpectedly impressive, the model has poor robustness. hence, the additional pre-trained model is introduced to solve the robustness problem. the latest model debertav3 @xcite has outstanding performance on cross-linguistic tasks, which outperforms bert and deberta on many tasks. the proposed model consists of two parts: the basic ernie-m model and the pre-trained model debertav3. the de-bertav3 model shares the same pre-trained data with ernie-m called xnli @xcite , which can improve the performance and robustness as well. the debertav3 model is trained independently, which has significant improvement for english but somehow brought no improvement for other languages. based on the above conclusion, we employ the debertav3 model for english-task only. to better understand the effectiveness of the proposed model, we started a bunch of analyses. the first problem is the data-set limitation. two additional datasets were imported, i.e., the translated dataset from google translation which is translated from three languages, and the xnli dataset. however, larger datasets don't lead to better performance. we compared the performance of the ernie-m model on four sets of data: the given data, the given data with translated data, the given data with xnli augmentation, and the given data with both the translated data and xnli data. we do the same experiments with the debertav3 model as well. the results show that the combination of ernie-m with all the three datasets and debertav3 with the given english data perform the best. ## related work multilingual model ernie-m proposes a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. there are two models in ernie-m which are cross-attention masked language mod-eling (camlm) and back-translation masked language modeling (btmlm). cross-attention masked language modeling (camlm) is to align cross-language semantic representations on parallel corpora. then, the multilingual representation is enhanced with transferability learned from parallel corpora. back-translation masked language modeling (btmlm) is trained to generate pseudo-parallel sentences from monolingual sentences. the generated pairs are then used as the input of the model to further align the cross-lingual semantics, thus enhancing the multilingual representation. debertav3 presents a new pre-trained language model, which improves the original de-berta model by replacing mask language modeling (mlm) with replaced token detection (rtd), a more sample-efficient pre-training task. they all come from an important field, multilingual models. since the related paper was published at the end of 2021, there are no similar tasks have been done and published. ## our approach in this section, we first introduce the methods to solving the multi-language problem and then present our work about improving the performance on uni-language. to extenuate over-fitting for a specific language, our team uses a multi-language ensemble learning strategy that includes a pre-trained language model and a multilingual language model. based on the approach above, it makes the learned representation generalizable across languages and improves the performance in finding the suitable taxonomic relations in two nominal arguments. ## experiments in this section, we first describe the dataset and our data preprocessing steps, and then we present the details of the experimental setup for subtask1. ## conclusion to solve the problem of judging whether the meaning of a sentence is self-consistent in multilingual language tasks, that is, the problem raised in task 3, we propose an ensemble model using ernie-m and debertav3, and regard this problem as a binary classification problem. furthermore, to solve the issue of the small dataset, we use various strategies, such as k-ford cross-validation, translating the dataset using different translators, and introducing an external dataset -xnli, a dataset commonly used in multilingual problems. in future efforts, we plan to further improve our model from these aspects. the first is to enrich the data, especially italian and french, to help the model learn better. the second is that we could train more models on standard fine-tuning, multi-step fine-tuning,