W ill_ G o at S em E val-2020 Task 3: An Accurate Model for Predicting the (Graded) Effect of Context in Word Similarity Based on BERT (2020)

Abstract: Natural Language Processing (NLP) has been widely used in the semantic analysis in recent years. Our paper mainly discusses a methodology to analyze the effect that context has on human perception of similar words, which is the third task of SemEval 2020. We apply several methods in calculating the distance between two embedding vector generated by Bidirectional Encoder Representation from Transformer (BERT). Our team will go won the 1st place in Finnish language track of subtask1, the second place in English track of subtask1.

## introduction computing the meaning difference between words in the semantic level is a task that has been widely discussed. in the area of natural language processing (nlp) like information retrieval (ir), there are many specific applications using similarity, such as text memorization @xcite ), text categorization @xcite ), text q&a @xcite , etc. the task3 of semeval-2020 1 focuses on the influence of context when humans perceive similar words @xcite . as we all know, polysemous words have different meanings in a totally different context, which the current translation system can recognize very well. however, many translation systems can't exactly predict the subtle variance on the meanings of words, which is also caused by a different context. task3 has two sub-tasks. in subtask1, we are required to predict the extent of change in scores of similarity between two words in different contexts by human annotators. in subtask2, we only predict the absolute score as is in the subtask1 rather than the difference in scores, and we would only discuss subtask1. our team uses different algorithms to calculate the distance between two embedding vectors generated by bert @xcite and defines it as the similarity. so we can get the change in similarity by subtraction between two distances. however, this methodology did not get exciting performance in the task evaluation, so we improve this by blending different bert models, which we will introduce later in section 3.2. ## related work there are many methods and models to estimate the similarity between long paragraphs. most of them treat it as a binary classification problem, hatzivassiloglou et al. @xcite compute the linguistic vector of features including primitive features and composite features, then they build criteria by feature vectors to classify paragraphs. as for similarity between short sentences, foltz et al. @xcite suggest a method that provides a characterization of the extent of semantic similarity between two adjacent short sentences by comparing their high-dimensional semantic vectors, which is also a latent semantic analysis (lsa) model. both lsa and hyperspace analogues to language (hal) @xcite are all corpus-based model, the latter one uses lexical co-occurrence to generate high-dimensional semantic vectors set, where words in this set can be represented as a vector or high-dimensional point so that their similarities can be measured by computing their distances. although computing similarity between words are less difficult than between texts, there still exist some sophisticated problems. similarity between words is not only in morphology but more significantly in semantic meaning. the first step of reckoning the similarity between words is using word2vec @xcite , which is a group of corpus-based models to generate word embedding, and mainly utilizes two architectures: continuous bag-of-words (cbow) and continuous skip-gram. in the cbow model, the distributed representations of context are made as input to the model and predict the center words, while the skip-gram model uses the center words as its input and predict the context, which predicts one word in many times to produce several context words. therefore, the skip-gram model can learn efficiently from context and performs better than the cbow model, but it takes much more consumption in training time than the cbow model. but since hierarchical softmax and negative sampling @xcite were proposed to optimize the skip-gram model when training large-scale data. word2vec cannot be used for computing similarity between polysemous words because it generates only one vector for a word, while embedding from language model (elmo) @xcite inspired by semi-supervised sequence tagging @xcite can handle this issue. elmo is consist of bidirectional lstm @xcite , which makes the elmo have an understanding of both next and previous word, it obtains contextualized word embedding by weight summation over the output of hidden layers. compared with the lstm used in elmo, bidirectional encoder representation from transformer (bert) @xcite is a stack of transformer encoder @xcite , which can be computed in parallel ways and save much time in training. there are two bert versions with different size, one is bert base, which has 12 encode layers with 768 hidden units and 12 attention heads, and the other is bert large, which has 24 encode layers with 1024 hidden units and 16 attention heads, achieved state-of-the-art results according to that paper.