Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information (2024)

Abstract: This paper addresses the problem of word sense induction (WSI) via clustering of word embeddings. It starts from the hypothesis that contextualized word representations obtained from pre-trained language models (LMs), while being a valuable source for WSI, encode more information than what is necessary for the identification of word senses and some of this information affect the performance negatively in unsupervised settings. We investigate whether using contextualized representations that are invariant to these ‘nuisance features’ can increase WSI performance. For this purpose, we propose an adaptation of the adversarial training framework proposed by Jaiswal et al. (2020) to erase specific information from the representations of LMs, thereby creating feature-invariant representations. We experiment with erasing (i) morphological and (ii) syntactic features. The results of subsequent clustering for WSI show that these features indeed act like noise: Using feature-invariant representations, compared to using the original representations, increases clustering-based WSI performance. Furthermore, we provide an in-depth analysis of how the information about the syntactic and morphological features of words relate to and affect WSI performance.

## introduction words in their different senses occur in different contexts. contextualized word representations obtained from transformer based pre-trained language models (lms) such as bert (devlin et al., 2019) are especially suitable for word sense disambiguation (wsd) because they capture the sentential context of a word and thereby oftentimes allow to distinguish different senses of a word. they have indeed been successfully used for wsd in recent work (hadiwinoto et al., 2019; loureiro et al., 2021; vandenbussche et al., 2021). however, in both unsupervised wsd, where the goal is to identify the instances of a specific sense, and word sense induction (wsi), which allows the discovery of novel senses, using the lm representations alone does not yield satisfactory results (pilehvar and camacho-collados, 2019; samih and kallmeyer, 2023). in both, similarity of the representations plays a crucial role, and this similarity is determined by many features. indeed, the contextualized representation of a word usually encodes a wide range of linguistic information about the word in its context, such as its syntactic function, its morphological properties, its position, its casing, and the identity of its neighbouring words (sajjad et al., 2022). however, most of the encoded information that determines the similarity of the representations is not relevant to word senses (yavas, 2024). note that this is not a problem for supervised wsd, as a supervised model can learn to ignore those features that are not discriminative for word senses. building on these insights, we focus on wsi and aim at investigating the relationship between specific types of information encoded in contextualized representations of lms and wsi performance. concretely, we examine whether erasing certain information from the representations of lms can lead to an increase in performance for a simple clustering-based wsi system. our investigation examines two types of information that have been observed to affect the word sense clustering performance negatively. yavas (2024) have shown that in word sense clustering with bert representations on semcor (miller et al., 1993), word instances are frequently clustered together based on the similarities between their morphological and syntactic features (more specifically, syntactic role of the word) rather than their semantic similarities. for example, past tense instances of a specific verb, or all instances of a specific noun occurring as direct objects, are clustered together. we adapt the adversarial training framework of jaiswal et al. (2020) in order to train a forget-gate that erases information from the representations of lms, resulting in feature-invariant representations. we experiment with the bert model and create feature-invariant representations for both of the above-mentioned types of features (morphological and syntactic). finally, we evaluate the performance of wsi on semcor with different featureinvariant representations, comparing them to the original word representations obtained from bert. furthermore, we conduct an in-depth analysis of how the information about the syntactic and morphological features of words relate to and affect wsi performance. @xcite our results show that words' morphological and syntactic features indeed act like noise that negatively affects clustering performance and syntaxand morphology-invariant representations are better suited to wsi than the original bert representations. furthermore, a more detailed analysis of the relation between these information types and wsi performance shows that even though syntactic features are more correlated to word senses than morphological features are, they still affect the wsi performance negatively overall. this paper makes several contributions. first, we propose an adaptation of the framework proposed by jaiswal et al. (2020) to erase unwanted information from the representations of lms. secondly, we use this method to generate syntaxand morphology-invariant representations from the word representations of the bert model and achieve better performance in clustering-based wsi. lastly, we provide an in-depth analysis of how the morphological and syntactic features of words affect wsi performance. the paper is structured as follows: we review related work in section 2, then we introduce our framework for creating feature-invariant representations in section 3 and report the results of the creation process. finally in section 4, we report the experiments on wsi with an analysis of the relation between the information types and wsi performance. in their detailed qualitative analysis, yavas (2024) have shown that word sense clustering with bert's representations on semcor is heavily and negatively affected by information encoded in the representations from the sentence context that is insignificant to wsd, such as some morphological features of the words, their syntactic role, the presence of some punctuation marks and function words in the sentence (e.g. 'not', 'then', etc.). in the present study, we aim to investigate whether the effects of some of these features can be controlled and whether doing so can increase performance in wsi on the same dataset. similar effects have been found in lexical semantic change detection. laicher et al. (2021) have observed that bert representations are influenced by the orthographic form of words. consequently, this affects how the representations are clustered. they have shown that removal of this bias increases the clustering performance. in order to do so, they preprocess the input data by lemmatizing the target word in each sentence before feeding it to the model. ## creating feature-invariant representations via adversarial training in order to obtain contextualized representations that are invariant to certain features, we propose to add a forget-gate on top of a pre-trained lm. the forget-gate applies a nonlinear transformation and thereby selectively removes the unwanted information from the original contextualized representations. it is trained as a part of an adversarial model inspired by jaiswal et al. (2020). concretely, we train two forget-gates to create feature-invariant representations for syntactic and morphological features. we will refer to the respective resulting representations as syntax-invariant, and morphologyinvariant representations. ## word sense induction performance our aim is to investigate whether using featureinvariant contextualized representations can improve wsi performance. for this purpose, we compare the performance on wsi of three variants of the same system, respectively using three different representations; 1) the original contextualized representations of the bert model, 2) syntax-invariant, and 3) morphology-invariant contextualized representations, where the latter two are obtained by applying our trained forget-gates f g m and f g s to the bert representations. furthermore, we provide a detailed analysis of the relation between the morphological and syntactic features of words and wsi and how the erasure of this information affects the wsi performance. ## conclusion we adapt the framework proposed by jaiswal et al. (2020) in order to erase specific information from the representations of lms. with this method, we create two types of representations from bert embeddings: invariant to either (i) morphological features or (ii) syntactic features. our results show that the resulting feature-invariant representations are more suitable for the wsi task. furthermore, we show that even though some syntactic features provide valuable information for wsi, both types of features introduce noise that, overall, negatively impacts the performance of clustering-based wsi.