Multi-sense Language Modelling (2022)

Abstract: The effectiveness of a language model is influenced by its token representations, which must encode contextual information and handle the same word form having a plurality of meanings (polysemy). Currently, none of the common language modelling architectures explicitly model polysemy. We propose a language model which not only predicts the next word, but also its sense in context. We argue that this higher prediction granularity may be useful for end tasks such as assistive writing, and allow for more a precise linking of language models with knowledge bases. We find that multi-sense language modelling requires architectures that go beyond standard language models, and here propose a localized prediction framework that decomposes the task into a word followed by a sense prediction task. To aid sense prediction, we utilise a Graph Attention Network, which encodes definitions and example uses of word senses. Overall, we find that multi-sense language modelling is a highly challenging task, and suggest that future work focus on the creation of more annotated training datasets.

## introduction any variant of language model, whether standard left-to-right, masked @xcite or bidirectional @xcite has to address the problem of polysemy: the same word form having multiple meanings, as seen in tab. 1. the meaning of a particular occurrence depends on the context, and all modern language modelling architectures from simple rnns @xcite to transformers @xcite use context-based representations. however, token representations in language models are not explicitly disambiguated. single-prototype embeddings, i.e., traditional word vectors, have a 1-to-1 correspondence with word forms. contextual embeddings change depending on the tokens in their context window, and are employed in recent models like ulmfit @xcite sentence meaning "john sat on the bank of the river and watched the currents" bank.n.01: sloping land, especially the slope beside a body of water "jane went to the bank to discuss the mortgage" bank.n.02: a financial institution that accepts deposits and channels the money into lending activities ruder, 2018), elmo @xcite , and all transformer architectures. however, even for contextual embeddings, polysemy is handled in an implicit, non-discrete way: the sense that a word assumes in a particular occurrence is unspecified. here, we propose the task of multi-sense language modelling, consisting of not only word, but also sense prediction. we conjecture that multisense language modelling would: another potential use would be to explore if such dictionary information could improve standard language modelling, and reduce the number of training data needed, relevant for e.g. language modelling for low-resource languages. consequently, our research objectives are to: as a sense inventory, we use wordnet 3.0 @xcite . the sense background knowledge is encoded in a dictionary graph, as shown in fig. 3 . when reading a word w, the model can rely on an additional input signal: the state of the node that represents w in the dictionary graph (the "global node"). node vectors in the graph are updated by a graph attention network @xcite . findings: we find that sense prediction is a significantly more difficult task than standard word prediction. a way to tackle it is to use a localized prediction framework, where the next sense depends on the prediction of the next word. the most successful model we identified for this uses a hard cut-off for the number of words considered (selectk, see § 4.4). the additional input signal from the dictionary graph provides only marginal improvements. for future work, we argue that multi-sense language modelling would benefit from larger sense-labelled datasets, possibly aided by wordnet super-sense information (categories like food, artifact, person, etc.) to deal with the excessively fine granularity of wordnet senses. ## related work we here briefly discuss relevant works that disambiguate between word senses to address polysemy. they can be grouped in three categories: 1) multiprototype embeddings not connected to a knowledge base; 2) supervised multi-sense embeddings based on a text corpus that only utilise a kb tangentially as the sense inventory; and 3) models that rely more substantially on features from a kb, like glosses or semantic relations. @xcite learn multi-prototype vectors by clustering word context representations. single-prototype embeddings are determined by 2 ff-nns with a margin objective on predicting the next word, a quasi-language modelling setting even if it utilises both the preceding and subsequent context. multi-sense skip-gram @xcite also defines senses as cluster centroids, measuring the cosine distance of the surrounding context @xmath0. @xcite use chinese restaurant processes to decide whether to create a new cluster, and also investigate the usefulness of multi-sense embeddings in downstream tasks such as semantic relatedness and pos tagging. @xcite ) create multi-prototype embeddings from bert, to address word similarity and relatedness tasks. every word w is associated with a set of occurrence embeddings, which are computed by averaging sub-word wordpiece tokens. senses are obtained by applying k-means clustering to the set. each bert layer contributes a different set, thus the authors found that middle layers are more relevant to the similarity task, the final layers to relatedness. supervised multi-sense embeddings other models rely on sense-label supervision in a text corpus. context2vec @xcite builds contextual word embeddings by applying a bilstm on text, and provides the option to create sense embeddings by using a sense-labelled corpus. @xcite frames wsd as a sequence learning problem, with the aim of finding sense labels for an input sequence. the training corpus is the same used in our work, semcor @xcite , and the core architecture is a bilstm that reads both the preceding and subsequent context. a bilstm is also employed in lstmembed @xcite , that obtains a sense-labelled training corpus by applying the babelfly @xcite sense tagger on the english wikipedia and other texts. ## architectures for sense prediction as previously described, multi-sense language modelling consists of the two tasks of standard language modelling and sense prediction, and we aim to output two probability distributions -for the next word token and the next sense token, respectively. ## discussion as shown in tab. 2, next-token prediction at the granularity of senses is a more difficult task than standard language modelling, due to operating on a larger vocabulary with a more extensive lowfrequency long tail. to try to overcome this obsta-cle, we proposed a localized prediction framework, finding that it is extremely reliant on the correctness of standard language model prediction. an argument can be made for developing better sense discrimination models that can work with a higher number of candidate senses, for instance those deriving from @xmath1. with this in mind, we observe that there are relatively few sense-labelled datasets. the datasets organised in ufsac format @xcite altogether contain 44.6m words, of which only 2.0m are annotated; in semcor, 29.4% of the tokens have a sense label. these datasets are available for english only, thus, studying the benefit of using dictionary resources for low-resource languages cannot currently be pursued until such corpora are created. as seen in tab. 3, the best results we managed to achieve are obtained by choosing among the senses of the most likely word. if a sense prediction method managed to reliably choose among a higher number of candidate senses, it would make the sense prediction task less dependant on achieving a good performance for the standard language modelling task. the question of what such a method could be remains open. it may be solved by investigating different wsd methods, and possibly by different ways of encoding the dictionary graph. moreover, one could expect that next-token prediction, both at the word and the sense level, would benefit from a more accurate standard language modelling architecture, possibly pre-trained on a larger corpus than wikitext-2. however, wikitext-2 was chosen to avoid overwhelming semcor's vocabulary, so this brings us back to the necessity of a larger sense-labelled corpus, that would also allow one to use an architecture different than a gru to obtain logits over the senses. including the input signal from the dictionary graph only results in marginal improvements on the sense prediction task. it should be noted that the quality of the input signal is limited by the quality of the sentence encodings for the wordnet glosses, used to initialise the graph nodes. sentence representations different from averaging fasttext embeddings may achieve better results. moreover, tuning the graph signal is surely possible, while outside of the scope of this first study of multi-sense language modelling: one could experiment with changing the size of the graph area, the number of hops and the variant of graph neural network used. ## conclusions this work constitutes the first study of multi-sense language modelling. we experiment with a localized prediction approach that predicts a word followed by a word sense; as well as learning sense representations from both its sentential context, and a sense dictionary, encoding the latter using a graph neural network. the experimental results highlight the difficulty of such a novel task. some could regard word senses as not fit to be discretized in a language modeling task, and best represented by flexible contextual embeddings like those of transformers. we believe that specifying discrete senses in language modeling could still be improved investigating three directions: for instance, creating word embeddings from bert by averaging the wordpiece-encoded tokens, or applying other word-sense disambiguation methods; 3) predicting wordnet supersenses: higher level categories such as food, artifact, person; this would avoid relying on the fine granularity of wordnet senses, solving a relatively simpler task that would still provide useful distinctions. future work on this task may view it as a test bed for researching word sense disambiguation, as a way of improving the precision of linking a language model to a knowledge base, or for applications such as assistive writing.