JUSTM asters at S em E val-2020 Task 3: Multilingual Deep Learning Model to Predict the Effect of Context in Word Similarity (2020)

Abstract: There is a growing research interest in studying word similarity. Without a doubt, two similar words in a context may considered different in another context. Therefore, this paper investigates the effect of the context in word similarity. The SemEval-2020 workshop has provided a shared task (Task 3: Predicting the (Graded) Effect of Context in Word Similarity). In this task, the organizers provided unlabeled datasets for four languages, English, Croatian, Finnish and Slovenian. Our team, JUSTMasters, has participated in this competition in the two subtasks: A and B. Our approach has used a weighted average ensembling method for different pretrained embeddings techniques for each of the four languages. Our proposed model outperformed the baseline models in both subtasks and acheived the best result for subtask 2 in English and Finnish, with score 0.725 and 0.68 respectively. We have been ranked the sixth for subtask 1, with scores for English, Croatian, Finnish, and Slovenian as follows: 0.738, 0.44, 0.546, 0.512.

## introduction identifying and measuring words similarity is considered the base step for spotting the similarity between sentences or documents. word similarity is classified into three types: the first type is lexical similarity @xcite , which depends on the symmetry of the character sequence; the second one is semantic similarity @xcite , which depends on the literal meaning of the words; and the third type is pragmatics @xcite , which focuses on the meanings and the effects of words according to the context. task 3 in semeval 2020 @xcite is about finding the context effect on word meanings by giving two similar words in a different context. task 3 has two unsupervised subtasks, the first subtask asked to predict the degree and direction of change in the human annotator's scores of similarity between two words when presented in two different contexts. the second subtask asked to predict the human score of similarity for a pair of words within the same context. in addition, the task provided data for four different languages: english, croatian, finnish, and slovenian. in this paper, we have experimented with different embeddings techniques in each of the four languages to determine the best combination for each subtask. also, we propose a weighted ensembling for stateof-the-art embedding techniques to achieve a significant sense of representation of the words. this has been accomplished for the four languages to gain the highest result in both subtasks except the english language. besides, our proposed model has outperformed the other teams in subtask 2 for english and finnish, with the highest score of 0.725 and 0.68 respectively. this paper is organized as follows: related work will be shown in section 2. section 3 provides a brief description of task3 and the dataset. our proposed model, justmasters, is characterized in section 4. experiments and results are introduced in section 5. finally, the conclusion is provided in section 6. ## related work different similarity measurements are used to determine the similarity between texts in various ways. there are four known categories of text similarity measurements: string-based, knowledge-based, corpusbased, and hybrid-based @xcite . the researchers agreed that string-based similarity algorithms are the best tool for measuring the lexical similarity, and knowledge-based similarity algorithms are the best for the semantic similarity. in order to estimate the score of similarity between sentences, researchers in @xcite applied manhattan long short-term memory (malstm) with support vector machine classifier by using 300-dimensions word2vec as word embedding. moreover, to capture the semantic similarity between fixed-length texts, a group of researchers @xcite proposed a hybrid technique that consists of tf-idf @xcite and word embedding. tf-idf technique performs well with logical similarity, while the word embedding performs well with semantic similarity, so they combined the previous technique to classify the texts using euclidean distance. other researchers @xcite used a novel combination of word embeddings, wordnet, and the concept dictionary 4lang to measure the semantic similarity of word pairs. the authors of @xcite presented a novel approach consisted of bi-lstm and pairwise word interaction modeling techniques to compare each word with the overall sentence. on the other hand, for traditional word embeddings, other researchers @xcite showed that the best values for the word2vec features are 9 as window size and 300 as vector dimensions configuration. for determining the sense of the words based on the contexts, the authors of @xcite ) proposed a model called sensebert, which predicts the sense in the semantic level of the missing words based on self-predictions of wordnet supersense @xcite . as well for word in context task, sensebert was attaining a state of the art result. for semantic relationships in semantic frame induction, a group of researchers @xcite proposed a combined approach that employed two different types of vector representations: dense representations from bert hidden layers of a masked language model, and sparse representations based on substitutes for the target word in the context. ## task description taks 3 in semeval 2020 consisted of two subtasks: data description: task 3 dataset is called cosimlex @xcite . it was released in four languages: english, croatian, finnish, and slovenian. the english dataset contains pairs of words derived from simlex-999 @xcite , while for croatian, finnish, and slovenian the pairs are translated from english to each language. moreover, it contains two different contexts extracted from wikipedia, each context has a pair of words. the size of each language: 340 english pairs, 112 croatian pairs, 111 slovenian pairs, and 24 finnish pairs. context 1 the tornado was also the second costliest tornado in history at the time, but in the years since has been surpassed by several others if population changes over time are not considered. when costs are normalized for wealth and inflation, it ranks third today. the deadliest tornado in world history was the daultipur-salturia tornado in bangladesh on april 26, 1989, which killed approximately 1300 people. ## results in this section, we will talk about the results of each language individually. the scores are based on subtask 2. ## conclusion in this paper, we describe our proposed models for semeval task 3. the task is designed for multilingual datasets that include english, croatian, finnish, and slovenian datasets. we also presented the experiments of our model with each language. justmaster model achieved excellent results in the competition on the leaderboard for english and finnish languages. the model for the english language consists of a concatenation of bert embedding, roberta embedding, and the extracted features, which achieved a 0.725 score for subtask 2, and 0.738 for subtask 1. while for the finnish language, we adopted the justmasters model to concatenate the bert and traditional word embedding vectors that achieved 0.68 scores for subtask 2 and a 0.546 score for subtask 1. on the other hand, for slovenian and croatian languages, we implemented the justmasters model with the traditional word embeddings, which obtained 0.44 and 0.443 respectively for subtask 2 and 0.512 and 0.440 respectively for subtask 1. it is worth mentioning that the extracted features such as tfidf, pos, and other features from both words in each context increased the overall accuracy by 0.15%.