Hitachi at S em E val-2020 Task 3: Exploring the Representation Spaces of Transformers for Human Sense Word Similarity (2020)

Abstract: In this paper, we present our system for SemEval-2020 task 3, Predicting the (Graded) Effect of Context in Word Similarity. Due to the unsupervised nature of the task, we concentrated on inquiring about the similarity measures induced by different layers of different pre-trained Transformer-based language models, which can be good approximations of the human sense of word similarity. Interestingly, our experiments reveal a language-independent characteristic: the middle to upper layers of Transformer-based language models can induce good approximate similarity measures. Finally, our system was ranked 1st on the Slovenian part of Subtask1 and 2nd on the Croatian part of both Subtask1 and Subtask2.

## introduction in this paper, we describe our participation in semeval-2020 task 3: predicting the (graded) effect of context in word similarity @xcite . the goal of the task is to understand the effect of contexts on word similarity. the task is composed of two subtasks sharing inputs: we are given a word pair (i.e., two words) and two text snippets (hereafter "contexts") both including the word pair. for convenience of explanation, we describe subtask2 first. in subtask2, we predict two similarity scores between the word pair in the two given contexts. in subtask1, we predict the difference in the above two similarity scores. more detailed descriptions of the subtasks are give in section 3. in both subtasks, small labeled data is available for model development. thus, participants are required to build models in an unsupervised manner. we formulated the tasks as the exploration of similarity measures induced in the hidden layer word representation space of pre-trained transformer @xcite based language models. our expectation is that contextualized representation of transformers could induce context dependent similarity measures that approximate the human perception. experimental results show that better approximations of word sense similarity can be induced in the middle to upper layers of transformers for most languages. as a result, our system was ranked 1st on the slovenian part of subtask 1 and 2nd on the croatian part of both subtask 1 and subtask 2. ## background capturing the similarity between words has been considered to be one of the fundamental tasks in natural language processing research because it is strongly related to many research fields such as text search, entailment recognition, and information extraction. recent work has been aimed at predicting the similarity of a given word pair @xcite and considers the similarity of word meanings that does not consider the effects of their contexts. word-sense disambiguation (wsd) @xcite , which is a lexicographical approach to the representation of word senses, aims at selecting an appropriate sense for a given word from wordspecific sense candidates. alongside with these two lines of the work, @xcite extended context text sim score c1 her prison cell was almost an improvement over her room at the last hostel. ## task formalization the task includes two subtasks, both of which aim at capturing the effect of context on similarities of word pairs. each subtask has four "sub-subtasks", each of those corresponding to each of four languages, namely, english, finnish, hungarian, and slovenian. as shown in given w and c, subtask 1 aims at predicting @xmath3, which expresses the change in similarity scores caused by contexts. the metric of subtask 1 is the pearson correlation coefficient between gold labels and predictions. due to the translational and scale invariance of pearson correlation, we can use the [-1, +1] range instead of @xcite . using this same input for subtask 1, subtask 2 aims at predicting s 1 and s 2 directly. the evaluation metric is the uncentered pearson correlation coefficient; thus, we can use any range in r as well. we take a two-stage approach; (i) we first solve subtask 2 by predicting s 1 and s 2 directly, and (ii) we second solve subtask 1 by calculating d from the predicted s 1 and s 2 . ## model as we mentioned in the above, we explore similarity measures in transformer's representation space exhaustively, which will represent the human sense of word similarity well. we introduce a cosine similarity-based measure and then take a "layer-wise" exploration strategy. ## conclusion in this paper, we proposed a model for the task of capturing the effects of context on word similarity. we employed similarity measures induced by the hidden layer representation vectors of pre-trained transformer-based language models. we explored all the layers of the models to find the one that matches human perception the best. our experimental results show that the multi-layered self-attention mechanism of transformer-based language models successfully captures the human sense of context-dependent word similarity. the results also revealed a universal language characteristic, that is, for all the transformer-based language models, the middle to upper layers perform better on the task than the others.