Learning to Learn to Disambiguate: M eta-Learning for Few-Shot Word Sense Disambiguation (2020)

Abstract: The success of deep learning methods hinges on the availability of large training datasets annotated for the task of interest. In contrast to human intelligence, these methods lack versatility and struggle to learn and adapt quickly to new tasks, where labeled data is scarce. Meta-learning aims to solve this problem by training a model on a large number of few-shot tasks, with an objective to learn new tasks quickly from a small number of examples. In this paper, we propose a meta-learning framework for few-shot word sense disambiguation (WSD), where the goal is to learn to disambiguate unseen words from only a few labeled instances. Meta-learning approaches have so far been typically tested in an N-way, K-shot classification setting where each task has N classes with K examples per class. Owing to its nature, WSD deviates from this controlled setup and requires the models to handle a large number of highly unbalanced classes. We extend several popular meta-learning approaches to this scenario, and analyze their strengths and weaknesses in this new challenging setting.

## introduction natural language is inherently ambiguous, with many words having a range of possible meanings. word sense disambiguation (wsd) is a core task in natural language understanding, where the goal is to associate words with their correct contextual meaning from a pre-defined sense inventory. wsd has been shown to improve downstream tasks such as machine translation @xcite and information retrieval @xcite . however, it is considered an ai-complete problem @xcite -it requires an intricate understanding of language as well as real-world knowledge. approaches to wsd typically rely on (semi-) supervised learning @xcite @xcite or are knowledge-based @xcite @xcite . while supervised methods generally outperform the knowledge-based ones @xcite , they require data manually annotated with word senses, which are expensive to produce at a large scale. these methods also tend to learn a classification model for each word independently, and hence may perform poorly on words that have a limited amount of annotated data. yet, alternatives that involve a single supervised model for all words @xcite still do not adequately solve the problem for rare words @xcite . humans, on the other hand, have a remarkable ability to learn from just a handful of examples @xcite . this inspired researchers to investigate techniques that would enable machine learning models to do the same. one such approach is transfer learning @xcite , which aims to improve the models' data efficiency by transferring features between tasks. however, it still fails to generalize to new tasks in the absence of a considerable amount of task-specific data for fine-tuning @xcite . meta-learning, known as learning to learn @xcite @xcite , is an alternative paradigm that draws on past experience in order to learn and adapt to new tasks quickly: the model is trained on a number of related tasks such that it can solve unseen tasks using only a small number of training examples. a typical meta-learning setup consists of two components: a learner that adapts to each task from its small training data; and a meta-learner that guides the learner by acquiring knowledge that is common across all tasks. meta-learning has recently emerged as a promising approach to few-shot learning. it has achieved success in computer vision @xcite @xcite and reinforcement learning @xcite @xcite . it has also recently made its way into nlp, and has been applied to machine translation @xcite , relation @xcite ) and text @xcite classification, and sentence-level semantic tasks @xcite . in this paper, we present the first meta-learning approach to wsd. we propose models that learn to rapidly disambiguate new words from only a few labeled examples. owing to its nature, wsd exhibits inter-word dependencies within sentences, has a large number of classes, and inevitable class imbalances; all of which present new challenges compared to the controlled setup in most current meta-learning approaches. to address these challenges we extend three popular meta-learning algorithms to this task: prototypical networks @xcite , model-agnostic meta-learning (maml) @xcite and a hybrid thereof -protomaml @xcite . we investigate meta-learning using three underlying model architectures, namely recurrent networks, multi-layer perceptrons (mlp) and transformers @xcite , and experiment with varying number of sentences available for task-specific fine-tuning. we evaluate the model's rapid adaptation ability by testing on a set of new, unseen words, thus demonstrating its ability to learn new word senses from a small number of examples. since there are no few-shot wsd benchmarks available, we create a few-shot version of a publicly available wsd dataset. we release our code as well as the scripts used to generate our few-shot data setup to facilitate further research. 1 2 related work ## task and dataset we treat wsd as a few-shot word-level classification problem, where a sense is assigned to a word given its sentential context. as different words may have a different number of senses and sentences may have multiple ambiguous words, the standard setting of n -way, k-shot classification does not hold in our case. specifically, different episodes can have a different number of classes and a varying number of examples per class -a setting which is more realistic @xcite . dataset we use the semcor corpus @xcite manually annotated with senses from the new oxford american dictionary by @xcite foot_0 . with 37, 176 annotated sentences, this is one of the largest sense-annotated english corpora. we group the sentences in the corpus according to which word is to be disambiguated, and then randomly divide the words into disjoint meta-train, meta-validation and meta-test sets with a 60:15:25 split. a sentence may have multiple occurrences of the same word, in which case we make predictions for all of them. we consider four different settings with the support set size |s| = 4, 8, 16 and 32 sentences. the number of distinct words in the meta-training / meta-test sets is 985/270, 985/259, 799/197 and 580/129 respectively. the detailed statistics of the resulting dataset are shown in appendix a.1. training episodes in the meta-training set, both the support and query sets have the same number of sentences. our initial experiments using one word per episode during meta-training yielded poor results due to an insufficient number of episodes. to overcome this problem and design a suitable metatraining setup, we instead create episodes with multiple annotated words in them. specifically, each episode consists of r sampled words {z j } r @xmath0, ν(z j )) senses for each of those words, where ν(z j ) is the number of senses for word z j . therefore, each task in the meta-training set is the disambiguation of r words between up to |s| senses. we set @xmath1. sentences containing these senses are then sampled for the support and query sets such that the classes are as balanced as possible. for example, for |s| = 8, we first choose 4 words and 2 senses for each, and then sample one sentence for each word-sense pair. the labels for the senses are shuffled across episodes, i.e., one sense can have a different label when sampled in another episode. this is key in meta-learning as it prevents memorization @xcite . the advantage of our approach for constructing meta-training episodes is that it allows for generating a combinatorially large number of tasks. herein, we use a total number of 10, 000 meta-training episodes. evaluation episodes for the meta-validation and meta-test sets, each episode corresponds to the task of disambiguating a single word. while splitting the sentences into support and query sets, we ensure that senses in the query set are present in the support set. furthermore, we only consider words with two or more senses in their query set. the distribution of episodes across different number of senses is shown in appendix a.1. note that, unlike the meta-training tasks, our meta-test tasks represent a natural data distribution, therefore allowing us to test our models in a realistic setting. ## methods our models consist of three components: an encoder that takes the words in a sentence as input and produces a contextualized representation for each of them, a hidden linear layer that projects these representations to another space, and an output linear layer that produces the probability distribution over senses. the encoder and the hidden layer are shared across all tasks -we denote this block as f θ with shared parameters θ. the output layer is randomly initialized for each task t i (i.e. episode) -we denote this as g φ i with parameters φ i . θ is meta-learned whereas φ i is independently learned for each task. ## discussion our results demonstrate that meta-learning outperforms the corresponding models trained in a non-episodic fashion when applied in a few-shot learning setting -a finding consistent for all |s| setups. using the bert-based models, we obtain up to 72% average macro f1 score with as few as 4 examples, and closely approach the reported state-of-the-art performance 3 with |s| = {16, 32}. the success of meta-learning is particularly evident with glove+gru. glove embeddings are sense-agnostic and yet, protonet, protofomaml and protomaml approach the performance of some elmo-based models, which enjoy the benefit of contextualization via large-scale pretraining. although contextualized representations from elmo and bert already contain information relevant to our task, integrating them into a metalearning framework allows these models to substantially improve performance. to illustrate the advantage that meta-learning brings, we provide example t-sne visualizations (van der @xcite of the original elmo embeddings and those generated by protonet based on elmo (figure 4 ). the representations from protonet are more accurately clustered with respect to the senses than the original elmo representations. protonet thus effectively learns to disambiguate new words, i.e. separate the senses into clusters, thereby improving upon elmo embeddings. we provide further t-sne visualizations in appendix a.6. the success of protonet and protofomaml can be in part attributed to the nature of the problem -wsd lends itself well to modeling approaches 3 not a direct comparison due to different data splits. our experiments further highlight the weakness of fomaml when applied beyond the n -way, k-shot setting. this may be due to the fact that the number of "new" output parameters in each episode is much greater than the number of support examples. informed output layer initialization in proto(fo)maml is therefore important for effective learning in such scenarios. a similar problem with fomaml is also pointed out by @xcite , who design a differentiable parameter generator for the output layer. ## conclusion few-shot learning is a key capability for ai to reach human-like performance. the development of meta-learning methods is a promising step in this direction. we demonstrated the ability of metalearning to disambiguate new words when only a handful of labeled examples are available. given the data scarcity in wsd and the need for few-shot model adaptation to specific domains, we believe that meta-learning can yield a more general and widely applicable disambiguation model than traditional approaches. interesting avenues to explore further would be a generalization of our models to disambiguation in different domains, to a multilingual scenario or to an altogether different task. |s| = 4 ward, delicate, jam, lose, year, bounce, haul, introduce, guard, suffer |s| = 8 bad, work, give, clear, settle, bloom, draw, check, break, gather |s| = 16 move, appearance, in, green, fix, establishment, note, drag, cup, bounce |s| = 32 independent, gather, north, square, do, bond, proper, pull, problem, language