Meta-Learning with Variational Semantic Memory for Word Sense Disambiguation (2021)

Abstract: A critical challenge faced by supervised word sense disambiguation (WSD) is the lack of large annotated datasets with sufficient coverage of words in their diversity of senses. This inspired recent research on few-shot WSD using meta-learning. While such work has successfully applied meta-learning to learn new word senses from very few examples, its performance still lags behind its fully-supervised counterpart. Aiming to further close this gap, we propose a model of semantic memory for WSD in a meta-learning setting. Semantic memory encapsulates prior experiences seen throughout the lifetime of the model, which aids better generalization in limited data settings. Our model is based on hierarchical variational inference and incorporates an adaptive memory update rule via a hypernetwork. We show our model advances the state of the art in few-shot WSD, supports effective learning in extremely data scarce (e.g. one-shot) scenarios and produces meaning prototypes that capture similar senses of distinct words.

## introduction disambiguating word meaning in context is at the heart of any natural language understanding task or application, whether it is performed explicitly or implicitly. traditionally, word sense disambiguation (wsd) has been defined as the task of explicitly labeling word usages in context with sense labels from a pre-defined sense inventory. the majority of approaches to wsd rely on (semi-)supervised learning @xcite @xcite @xcite and make use of training corpora manually annotated for word senses. typically, these methods require a fairly large number of annotated training examples per word. this problem is exacerbated by the dramatic imbalances in sense frequencies, which further increase the need for annotation to capture a diversity of senses and to obtain sufficient training data for rare senses. this motivated recent research on few-shot wsd, where the objective of the model is to learn new, previously unseen word senses from only a small number of examples. @xcite presented a meta-learning approach to few-shot wsd, as well as a benchmark for this task. meta-learning makes use of an episodic training regime, where a model is trained on a collection of diverse few-shot tasks and is explicitly optimized to perform well when learning from a small number of examples per task @xcite @xcite . @xcite have shown that meta-learning can be successfully applied to learn new word senses from as little as one example per sense. yet, the overall model performance in settings where data is highly limited (e.g. one-or two-shot learning) still lags behind that of fully supervised models. in the meantime, machine learning research demonstrated the advantages of a memory component for meta-learning in limited data settings @xcite @xcite . the memory stores general knowledge acquired in learning related tasks, which facilitates the acquisition of new concepts and recognition of previously unseen classes with limited labeled data @xcite . inspired by these advances, we introduce the first model of semantic memory for wsd in a meta-learning setting. in meta-learning, prototypes are embeddings around which other data points of the same class are clustered @xcite . our semantic memory stores prototypical representations of word senses seen during training, generalizing over the contexts in which they are used. this rich contextual information aids in learning new senses of previously unseen words that appear in similar contexts, from very few examples. the design of our prototypical representation of word sense takes inspiration from prototype theory @xcite , an established account of category representation in psychology. it stipulates that semantic categories are formed around prototypical members, new members are added based on resemblance to the prototypes and category membership is a matter of degree. in line with this account, our models learn prototypical representations of word senses from their linguistic context. to do this, we employ a neural architecture for learning probabilistic class prototypes: variational prototype networks, augmented with a variational semantic memory (vsm) component @xcite . unlike deterministic prototypes in prototypical networks @xcite , we model class prototypes as distributions and perform variational inference of these prototypes in a hierarchical bayesian framework. unlike deterministic memory access in memory-based meta-learning @xcite , we access memory by monte carlo sampling from a variational distribution. specifically, we first perform variational inference to obtain a latent memory variable and then perform another step of variational inference to obtain the prototype distribution. furthermore, we enhance the memory update of vanilla vsm with a novel adaptive update rule involving a hypernetwork @xcite that controls the weight of the updates. we call our approach β-vsm to denote the adaptive weight β for memory updates. we experimentally demonstrate the effectiveness of this approach for few-shot wsd, advancing the state of the art in this task. furthermore, we observe the highest performance gains on word senses with the least training examples, emphasizing the benefits of semantic memory for truly few-shot learning scenarios. our analysis of the meaning prototypes acquired in the memory suggests that they are able to capture related senses of distinct words, demonstrating the generalization capabilities of our memory component. we make our code publicly available to facilitate further research. 1 2 related work word sense disambiguation knowledge-based approaches to wsd @xcite 1 https://github.com/ydu-uva/vsm_wsd 2014; @xcite rely on lexical resources such as wordnet @xcite and do not require a corpus manually annotated with word senses. alternatively, supervised learning methods treat wsd as a word-level classification task for ambiguous words and rely on sense-annotated corpora for training. early supervised learning approaches trained classifiers with hand-crafted features @xcite and word embeddings @xcite as input. @xcite proposed a benchmark for wsd based on the semcor corpus @xcite and found that supervised methods outperform the knowledgebased ones. neural models for supervised wsd include lstm-based @xcite classifiers @xcite @xcite , nearest neighbour classifier with elmo embeddings @xcite , as well as a classifier based on pretrained bert representations @xcite . recently, hybrid approaches incorporating information from lexical resources into neural architectures have gained traction. glossbert @xcite fine-tunes bert with word-net sense definitions as additional input. ewise @xcite learns continuous sense embeddings as targets, aided by dictionary definitions and lexical knowledge bases. @xcite present a semi-supervised approach for obtaining sense embeddings with the aid of a lexical knowledge base, enabling wsd with a nearest neighbor algorithm. by further exploiting the graph structure of wordnet and integrating it with bert, ewiser @xcite achieves the current state-of-the-art performance on the benchmark by @xcite -an f1 score of 80.1%. unlike few-shot wsd, these works do not finetune the models on new words during testing. instead, they train on a training set and evaluate on a test set where words and senses might have been seen during training. meta-learning meta-learning, or learning to learn @xcite @xcite , is a learning paradigm where a model is trained on a distribution of tasks so as to enable rapid learning on new tasks. by solving a large number of different tasks, it aims to leverage the acquired knowledge to learn new, unseen tasks. the training set, referred to as the meta-training set, consists of episodes, each corresponding to a distinct task. every episode is further divided into a support set containing just a handful of examples for learning the task, and a query set containing examples for task evaluation. in the meta-training phase, for each episode, the model adapts to the task using the support set, and its performance on the task is evaluated on the corresponding query set. the initial parameters of the model are then adjusted based on the loss on the query set. by repeating the process on several episodes/tasks, the model produces representations that enable rapid adaptation to a new task. the test set, referred to as the meta-test set, also consists of episodes with a support and query set. the meta-test set corresponds to new tasks that were not seen during meta-training. during metatesting, the meta-trained model is first fine-tuned on a small number of examples in the support set of each meta-test episode and then evaluated on the accompanying query set. the average performance on all such query sets measures the few-shot learning ability of the model. metric-based meta-learning methods @xcite @xcite learn a kernel function and make predictions on the query set based on the similarity with the support set examples. model-based methods @xcite employ external memory and make predictions based on examples retrieved from the memory. optimization-based methods @xcite @xcite directly optimize for generalizability over tasks in their training objective. meta-learning has been applied to a range of tasks in nlp, including machine translation @xcite , relation classification @xcite ), text classification @xcite , hypernymy detection @xcite , and dialog generation @xcite . it has also been used to learn across distinct nlp tasks @xcite as well as across different languages @xcite . @xcite show that meta-learning during self-supervised pretraining of language models leads to improved fewshot generalization on downstream tasks. @xcite propose a framework for few-shot word sense disambiguation, where the goal is to disambiguate new words during meta-testing. meta-training consists of episodes formed from multiple words whereas meta-testing has one episode corresponding to each of the test words. they show that prototype-based methods -prototypical networks @xcite and first-order protomaml @xcite ) -obtain promising results, in contrast with model-agnostic meta-learning (maml) @xcite . memory-based models memory mechanisms @xcite @xcite have recently drawn increasing attention. in memory-augmented neural network @xcite , given an input, the memory read and write operations are performed by a controller, using soft attention for reads and least recently used access module for writes. meta network @xcite uses two memory modules: a key-value memory in combination with slow and fast weights for one-shot learning. an external memory was introduced to enhance recurrent neural network in @xcite , in which memory is conceptualized as an adaptable function and implemented as a deep neural network. semantic memory has recently been introduced by @xcite for few-shot learning to enhance prototypical representations of objects, where memory recall is cast as a variational inference problem. in nlp, @xcite use content and location-based neural attention over external memory for aspect-level sentiment classification. @xcite use key-value memory for question answering on knowledge bases. mem2seq @xcite is an architecture for task-oriented dialog that combines attention-based memory with pointer networks @xcite . @xcite propose dynamic memory induction networks for few-shot text classification, which utilizes dynamic routing @xcite over a static memory module. episodic memory has been used in lifelong learning on language tasks, as a means to perform experience replay @xcite @xcite . ## task and dataset we treat wsd as a word-level classification problem where ambiguous words are to be classified into their senses given the context. in traditional wsd, the goal is to generalize to new contexts of word-sense pairs. specifically, the test set consists of word-sense pairs that were seen during train-ing. on the other hand, in few-shot wsd, the goal is to generalize to new words and senses altogether. the meta-testing phase involves further adapting the models (on the small support set) to new words that were not seen during training and evaluates them on new contexts (using the query set). it deviates from the standard n -way, k-shot classification setting in few-shot learning since the words may have a different number of senses and each sense may have different number of examples @xcite , making it a more realistic few-shot learning setup @xcite . dataset we use the few-shot wsd benchmark provided by @xcite . it is based on the semcor corpus @xcite , annotated with senses from the new oxford american dictionary by @xcite . the dataset consists of words grouped into meta-training, metavalidation and meta-test sets. the meta-test set consists of new words that were not part of metatraining and meta-validation sets. there are four setups varying in the number of sentences in the support set |s| = 4, 8, 16, 32. |s| = 4 corresponds to an extreme few-shot learning scenario for most words, whereas |s| = 32 comes closer to the number of sentences per word encountered in standard wsd setups. for |s| = 4, 8, 16, 32, the number of unique words in the meta-training / meta-validation / meta-test sets is 985/166/270, 985/163/259, 799/146/197 and 580/85/129 respectively. we use the publicly available standard dataset splits. 2 episodes the meta-training episodes were created by first sampling a set of words and a fixed number of senses per word, followed by sampling example sentences for these word-sense pairs. this strategy allows for a combinatorially large number of episodes. every meta-training episode has |s| sentences in both the support and query sets, and corresponds to the distinct task of disambiguating between the sampled word-sense pairs. the total number of meta-training episodes is 10, 000. in the meta-validation and meta-test sets, each episode corresponds to the task of disambiguating a single, previously unseen word between all its senses. for every meta-test episode, the model is fine-tuned on a few examples in the support set and its generalizability is evaluated on the query set. in contrast to 4 methods ## experiments and results experimental setup the size of the shared linear layer and memory content of each word sense is 64, 256, and 192 for glove+gru, elmo+mlp and bert respectively. the activation function of the shared linear layer is tanh for glove+gru and relu for the rest. the inference networks g φ (•) for calculating the prototype distribution and g ψ (•) for calculating the memory distribution are all three-layer mlps, with the size of each hidden layer being 64, 256, and 192 for glove+gru, elmo+mlp and bert. the activation function of their hidden layers is elu @xcite , and the output layer does not use any activation function. each batch during meta-training includes 16 tasks. the hypernetwork f β (•) is also a threelayer mlp, with the size of hidden state consistent with that of the memory contents. the linear layer activation function is relu for the hypernetwork. for bert and |s| = {4, 8}, λ @xmath0.001, λ @xmath1.0001 and learning rate is 5e-6; |s| = 16, λ @xmath2.0001, λ @xmath3.0001 and learning rate is 1e-6; |s| = 32, λ @xmath4.001, λ @xmath5.0001 and learning rate is 1e-5. hyperparameters for other models are reported in appendix a.2. all the hyperparameters are chosen using the meta-validation set. the number of slots in memory is consistent with the number of senses in the meta-training set -2915 for |s| = 4 and 8; 2452 for |s| = 16; 1937 for |s| = 32. the evaluation metric is the wordlevel macro f1 score, averaged over all episodes in the meta-test set. the parameters are optimized using adam @xcite . we compare our methods against several baselines and state-of-the-art approaches. the nearest neighbor classifier baseline (nearestneighbor) predicts a query example's sense as the sense of the support example closest in the word embedding space (elmo and bert) in terms of cosine distance. the episodic fine-tuning baseline (ef-protonet) is one where only meta-testing is performed, starting from a randomly initialized model. prototypical network (protonet) and proto-fomaml achieve the highest few-shot wsd performance to date on the benchmark of @xcite . ## analysis and discussion to analyze the contributions of different components in our method, we perform an ablation study by comparing protonet, vpn, vsm and β-vsm and present the macro f1 scores in ## conclusion in this paper, we presented a model of variational semantic memory for few-shot wsd. we use a variational prototype network to model the prototype of each word sense as a distribution. to leverage the shared common knowledge between tasks, we incorporate semantic memory into the probabilistic model of prototypes in a hierarchical bayesian framework. vsm is able to acquire longterm, general knowledge that enables learning new senses from very few examples. furthermore, we propose adaptive β-vsm which learns an adaptive memory update rule from data using a lightweight hypernetwork. the consistent new state-of-the-art performance with three different embedding functions shows the benefit of our model in boosting few-shot wsd. since meaning disambiguation is central to many natural language understanding tasks, models based on semantic memory are a promising direction in nlp, more generally. future work might investigate the role of memory in modeling meaning variation across domains and languages, as well as in tasks that integrate knowledge at different levels of linguistic hierarchy.