Ferryman at S em E val-2020 Task 3: Bert with TFIDF -Weighting for Predicting the Effect of Context in Word Similarity (2020)

Abstract: Word similarity is widely used in machine learning applications like searching engine and recommendation. Measuring the changing meaning of the same word between two different sentences is not only a way to handle complex features in word usage (such as sentence syntax and semantics), but also an important method for different word polysemy modeling. In this paper, we present the methodology proposed by team Ferryman. Our system is based on the Bidirectional Encoder Representations from Transformers (BERT) model combined with term frequency-inverse document frequency (TF-IDF), applying the method on the provided datasets called CoSimLex, which covers four different languages including English, Croatian, Slovene, and Finnish. Our team Ferryman wins the the first position for English task and the second position for Finnish in the subtask 1.

## introduction word similarity quantitatively measures semantic similarity of two words by comparing word vectors or word embedding, multi-dimensional meaning representations of two words. word embedding models can be trained and used to predict the value of word similarity, which are comparable to methods based on human judgments. however, most of these established methods are context-independent. context plays a significant role for people to understand the meaning of words from different perspectives. the meaning of an ambiguous word can be inferred by a certain context, which is known as contextual selection of senses. in a single sense, context can modify the meaning of a word by highlighting certain semantic traits and backgrounding others, which is called the contextual modulation of meaning. to fully utilize context and avoid treating words in isolation, a model that can reflect similarity judgement of words in context is required, which takes word sense, discrete effects and other factors of context into consideration @xcite . in this paper, we primarily focus on quantifying the similarity of two words in different contexts. first, literature surveys have been conducted to obtain the similarity scores of two words in two different contexts. then, the uncentered pearson correlation coefficient between the scores output by our model and the scores output by the human annotators are calculated. according to the pearson correlation coefficient, we can evaluate the predicted performance of our context-dependent embeddings model in human perception of similarity. we have pre-processed the data by replacing abbreviation and used tf-idf @xcite to enrich the attributes of the pair of words to be evaluated in the mask layer of the bert. then we fine-tune the bert model @xcite with multi-dropout for predicting the absolute similarity rating for each pair of words in each context. among all teams participating in cosimlex, our models ranks the 1st place out of 14 on subtask 1. ## related work the effect of context in word similarity is a cutting-edge topic, which has drawn attention from both industry and academic realm. a number of works have made important contributions in the past several years @xcite . the dataset for this competition has been explained in cosimlex(carlos santos armendariz, 2019), which is suitable for intrinsic evaluation of state-of-the-art contextual word embedding models. different embedding algorithms will drastically affect end results, so we need to choose one that will deliver the results we're hoping for. term frequency-inverse document frequency(tf-idf)(das, 2007) is a statistical measure that evaluates how relevant a word is to a document a collection of documents. the dimension reduction is also needed for the sparse and high-dimensional vectors, which are not conducive to calculation. both principal component analysis(pca) @xcite and singular value decomposition(svd) @xcite methods can be adopted to get the corresponding k-dimensional vectors. in order to catch the meaning of the same word in different sentences, one of the most effective strategies is to use computational methods to encoding sentence information containing specific word. word2vec @xcite ) is a two-layer neural network that processes text by vectorizing words. the input of word2vec is a text corpus and its output is a set of feature vectors that present words in that corpus. word2vec has two training methods. the first one is continuous bag-of-word model (cbow) @xcite and the second one is skip-gram @xcite . there are more ways to train a word embedding. global vectors for word representation(glove)(pennington j, 2014) is an unsupervised learning algorithm for obtaining vector representations for words. training is performed on aggregated global word-word co-occurence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space. both of word2vec and glove provide the same core output, a vector per word. the difference between the two models is that word2vec does incremental, "sparse" training of a neural network, by repeatedly iterating over a training corpus, but glove works to fit vectors to model a gaint word co-occurence matrix built from corpus. doc2vec(q. le, 2014) model is based on word2vec, with only adding another vector(paragrah id) to the input. more specifically, doc2vec is based on the cbow model, but instead of using just nearby words to predict the word, it also added another feature vector, which is document-unique. so when training the vectors w, the document vector d is trained as well, and in the end of training, it hold a numeric representation of the document. moreover, in order to deal with the situation that word2vec cannot be used to efficiently measure the change in similarity socres between two contexts, a conrtuction method based on word2vec and hownet @xcite has been proposed. this method uses the word2vec training corpus to obtain the word vector and then obtains the 10 words closet to the candidate word. then, the hownet method is used to determine the polarity by calculating the semantic similarity between the candidate word and the seed word. the experimental results show that the construction method has high accuracy and availability. models such as embeddings from language models(elmo)(peters and zettlemoyer, 2018) and bidirectional encoder representations from transformers(bert) @xcite are able to provide representations of words according to surrounding context, which considers not only of discrete differences in word sense but of the more graded effects of context. the generative pre-training(gpt-2) @xcite model uses conditional probability language modeling with a transformer neural network architecture that relies on self-attention mechanisms in lieu of recurrence or convolution. ## conclusion in this study, we report our systems for cosimlex. to solve the problem of context-dependent word embeddings, we preprocessed the data by replacing abbreviation and unfine-tuned a bert-based model to predict the similarity of two words in two different contexts.