BRUMS at S em E val-2020 Task 3: Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity (2020)

Abstract: This paper presents the team BRUMS submission to SemEval-2020 Task 3: Graded Word Similarity in Context. The system utilises state-of-the-art contextualised word embeddings, which have some task-specific adaptations, including stacked embeddings and average embeddings. Overall, the approach achieves good evaluation scores across all the languages, while maintaining simplicity. Following the final rankings, our approach is ranked within the top 5 solutions of each language while preserving the 1st position of Finnish subtask 2.

## introduction in natural language, meaning of a word has an influence by its surrounding or context words. this impact is mainly driven by the associated linguistic and cognitive phenomena @xcite . following these two phenomena, it was found that word meanings have a continuous nature in addition to the commonly known discrete nature. from the cognitive perspective, word meanings can be assigned based on the conceptual structures in human mind @xcite . therefore, meaning of a word can be varied according to the mental state of the reader which is triggered with the contact of context words. thus, word meanings are not just limited to a discrete nature. from the linguistic perspective, context words can modify the meaning of a word by contextual selection and contextual modulation @xcite . contextual selection identifies the meanings of polysemous words (i.e. words with multiple senses) such as 'cell', 'bank', etc. using the context words. in this case, the most appropriate meaning will be picked from a set of discrete senses. for an example, phrase 'prison cell' implies that the word 'cell' refers a room. contextual modulation modifies the meanings of single sense words by highlighting their characteristics. thus, unlike the contextual selection, modulation makes a continuous effect on the meaning and it is widely available, because majority of the words are general to a certain extent. for an example, 'butter' is a single sense word. but, if it appears in the phrase 'poured the butter', context words reveal that the mentioned butter is in liquid state. focusing on the above-mentioned facts, it is important to consider both discrete and continuous effects while predicting the meaning of words in natural language text. even though there is a continuous effect on meaning, majority of the previous research were only focused on the discrete effect. considering this limitation, semeval-2020 task 3 was designed by targeting the continuous (graded) effects of context. with the involvement of continuous effects, the goal of this task is predicting the effect of context in human perception of similarity. for an example, given the phrases: as participants of semeval-2020 task 3, we experimented the impact of contextualised word embeddings extracted by different architectures and learning methods on predicting the (graded) effect of context in word similarity. rather than focusing on the embeddings taken with default settings, we evaluated stacked embeddings, different parameter settings and average embeddings. further, we experimented the embeddings generated with improved known data rate. in this approach, model unknown words in the data set are replaced with model known words to support more effective embedding generation. the rest of this paper is organised as follows. section 2 describes the task including its subtasks and data sets. available methods related to this research are summarised under section 3 and our approaches are described in section 4. following these, results are mentioned in section 5 and paper is concluded with section 6. as the evaluation data set, cosimlex @xcite was used. this data set is newly created using human annotators, with the focus on graded effect on word similarity. as the first version, 340 english pairs, 112 croatian pairs, 111 slovenian pairs and 24 finnish pairs were released without human annotated scores to use with the evaluation phase of semeval-2020 task 3. among them, randomly selected 10 english pairs, 5 croatian pairs and 5 slovenian pairs were released including the human annotated scores as the practice data for participants to evaluate their models. ## related work effect of context was considered by previous research to predict the meaning of words and their similarity. in order to include wider context knowledge, @xcite introduced a neural network-based language model which considers both local and global contexts to learn word representations. as local context, sequence of words is considered and as global context, corresponding document is considered. to capture the multiple senses of a word, they suggested to cluster learned context word vectors into different meaning groups. in later research, there was a tendency to focus on sense embeddings. sense embeddings are vectors generated to represent different senses of words. @xcite suggested to use words in wordnet @xcite gloss contents to produce sense embeddings. to obtain vector representations of words, they used skip-gram model @xcite . for each polysemous word, a sense was picked by comparing the similarity between its context vector and available sense vectors. given a sentence, average of its word vectors was computed as the context vector. similar approach was suggested by another recent research using bert embeddings @xcite to incorporate more contextual features @xcite . in this approach, sense embedding generation was initiated using semcor corpus @xcite . using the sentences available in the corpus for each sense, bert embeddings were generated and average of those embeddings were taken as the sense embedding. to improve the sense coverage, wordnet's ontology and glosses also considered. given a sentence, generated bert embedding corresponding to a polysemous word was compared with sense embeddings to disambiguate it. using meaning groups and sense embeddings, above-mentioned approaches are only looking for discrete word meanings with the effect of context and we could not find any approach which considers both discrete and continuous effects. as we are aware of cosimlex is the first data set which is annotated by considering the continuous effect of context and semeval-2020 task 3 is the first shared task which is focused on predictions based on this effect. however, most of the available research are based on prediction-based word embeddings and among them contextualised word embeddings are found to be more capable in extracting word meanings based on the context. ## methodology this section describes the different approaches used to predict the (graded) effect of context in word similarity. since the similarity between a word pair is measured using the cosine similarity between word vectors, we experimented different methods to generate vector representations based on recently published contextualised word embedding models as further described in sections; 4.1 -4.4. all the implementations are done in python foot_0 . ## results this section contains the results obtained for both subtasks using the approaches described in section 4. we evaluated the suggested methods using practice data and used the best methods for submissions. the top three scores obtained by submissions on both tasks for all four languages and baseline scores are summarised under section 5.1 and 5.2. for all the reported transformer-based models, as the default parameter setting, concatenation of last four layers using first sub-token without scalar mix is used. any parameter change except the default setting is mentioned within the brackets after the model name. stacked models are mentioned using + symbol (e.g. model 1 + model 2 ) and phrase with ne is appended to model name if improved known data rate is used. as the baseline model for both tasks, the multilingual bert model released by gary lai foot_4 was used as informed by the task organisers. this model was trained using an uncased multilingual data set extracted from wikipedia. ## conclusion in this paper, we presented different approaches used for semeval-2020 task 3: graded word similarity in context. we mainly evaluated the recent contextualised word embedding models with different embedding generation techniques. depending on the differences between languages, we could not find any universal approach suitable for all languages to predict the effect of context in word similarity. for subtask 1: