GX at S em E val-2021 Task 2: BERT with Lemma Information for MCL - W i C Task (2021)

Abstract: This paper presents the GX system for the Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC) task. The purpose of the MCL-WiC task is to tackle the challenge of capturing the polysemous nature of words without relying on a fixed sense inventory in a multilingual and cross-lingual setting. To solve the problems, we use context-specific word embeddings from BERT to eliminate the ambiguity between words in different contexts. For languages without an available training corpus, such as Chinese, we use neuron machine translation model to translate the English data released by the organizers to obtain available pseudo-data. In this paper, we apply our system to the English and Chinese multilingual setting and the experimental results show that our method has certain advantages.

## introduction in recent years, contextual embeddings have drawn much attention. the approaches of calculating contextual embeddings include multi-prototype embeddings, sense-based and contextualized embeddings @xcite . however, it is not easy to evaluate such multiple embedding methods in one framework. @xcite present a large-scale word in context dataset to focus on the dynamic semantics of words. following and expanding them, the mcl-wic task @xcite performs a binary classification task that indicates whether the target word is used with the same or different meanings in the same language (multilingual data set) or across different languages (cross-lingual data set). besides, it is the first semeval task for word-in-context disambiguation @xcite . 1 reproducible code: https://github.com/yingwaner/bert4wic a typical solution to the problems is obtaining context-specific word embeddings, such as context2vec @xcite and bidirectional encoder representations from transformers (bert) @xcite . bert is designed to pre-train deep bidirectional representation in unlabeled texts by jointly conditioning in the left and right contexts of all layers. due to its powerful capabilities and easy deployment, we use bert as our major system and fine-tune on the training data released by the organizer to get the context-specific word embeddings. in this paper, we participate in the sub-task of multilingual settings in english and chinese. the organizer only provides english training data, and we fine-tune the pre-trained english bert model based on this data. for chinese tasks where no training set is available, we train a satisfactory neural machine translation (nmt) model to translate the english training set into chinese and then fine-tune the chinese bert model based on the pseudo-data. the experimental results show that our method achieves 82.7% in english multilingual setting and 76.7% in chinese multilingual setting. ## background in this section, we will briefly introduce the wordin-context task and the structure of bert for the sentence pair classification task. ## system overview systems proposed for both english and chinese multilingual settings were based on bert model @xcite with task-specific input modifications. we participate in the multilingual setting and divide the system into two parts according to the language: english setting and chinese setting. ## experimental setup in this section, we will describe the experimental settings for english and chinese in detail. ## results in this section, we will first report the main results of english and chinese multilingual setting and analyze the importance of all the factors in the system. then, we explore the probability of error for each part of speech. ## conclusion in this paper, we describe the gx system participating in the mcl-wic task. in order to obtain the general basic knowledge, we use the pre-trained bert model and then fine-tune it on the data released by the organizer. in order to further emphasize the relationship between sentence pairs and the importance of lemma, we introduce three new factors: lemma token, lemma segment embedding, and lemma embedding, and finally get better results. our system reaches 82.7% in english multilingual setting and 76.7% in chinese multilingual setting.