Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning (2021)

Abstract: Word embedding learning methods require a large number of occurrences of a word to accurately learn its embedding. However, out-of-vocabulary (OOV) words which do not appear in the training corpus emerge frequently in the smaller downstream data. Recent work formulated OOV embedding learning as a few-shot regression problem and demonstrated that meta-learning can improve results obtained. However, the algorithm used, model-agnostic meta-learning (MAML) is known to be unstable and perform worse when a large number of gradient steps are used for parameter updates. In this work, we propose the use of Leap, a meta-learning algorithm which leverages the entire trajectory of the learning process instead of just the beginning and the end points, and thus ameliorates these two issues. In our experiments on a benchmark OOV embedding learning dataset and in an extrinsic evaluation, Leap performs comparably or better than MAML. We go on to examine which contexts are most beneficial to learn an OOV embedding from, and propose that the choice of contexts may matter more than the meta-learning employed.

## introduction distributional methods for learning word embeddings require a sufficient number of occurrences of a word in the training corpus to accurately learn its embedding. even though the embeddings can be trained on raw text implying that an embedding for every word is obtained, in practice outof-vocabulary (oov) words do occur in the downstream applications embeddings are used, for example due to domain-specific terminology. nevertheless, oov words are often content words such as names which convey important information for downstream tasks; for example, drug names are key in the biomedical domain. however, the amount of downstream language data is typically much smaller than the corpus used for training word embeddings, thus methods that rely on distributional properties of words across large amounts of data perform poorly @xcite . researchers often assign oov words to random embeddings or to an "unknown" embedding, however these solutions fail to capture the distributional properties of words. zero-shot approaches @xcite @xcite attempt to predict the embeddings for oov words from their characters alone. these approaches rely on inferring the meaning of a word from its subword information, such as morphemes or wordpiece tokens used in bert @xcite . while this works well for many words, it performs poorly for names and words where morphology is not informative. given that an oov word occurs once, the chance of a second occurrence is much higher than the first @xcite . hence while oov words can be rare and not seen in training, it is reasonable to expect that a limited number of occurrences will be present in the data of a downstream application. few-shot approaches @xcite @xcite leveraged this to predict the embeddings for oov words from just a few contexts, often in conjunction with their morphological information. @xcite proposed an attention-based architecture for oov word embedding learning as a few-shot regression problem. the model is trained to predict the embedding of a word based on a few contexts and its character sequence. such a model is trained by simulating oov words in the training corpus, with their target embeddings provided by learning them on the same corpus. as oov words must have their embeddings inferred from contexts outside the training corpus, the authors show that using an adaptation of the model-agnostic meta-learning (maml) algorithm @xcite to adapt the model's parameters to the target domain improves the quality of the learned oov word embeddings. however, maml is known to be unstable due to the calculation of gradients requiring backpropagation through multiple instances of the model, as the learning process must be unrolled to calculate gradients with respect to the initial parameters @xcite . in practice, the learning process is often truncated to a small number of gradient steps, but has been shown to have a short-horizon bias @xcite , causing it to underperform. in this work we explore oov word embedding learning using leap @xcite , a meta-learning framework which takes into consideration the entire learning trajectory, not only the beginning and end points. each task is associated with a loss surface over the model's parameters on which the learning process travels, and the aim is to minimize the expected length of this process across tasks. leap also does not require backpropagation through the learning process, allowing it to adapt over a larger number of gradient steps and thus not suffering from the short-horizon bias that maml is prone to. we conduct an intrinsic evaluation of maml and leap on the dataset of @xcite which simulates oov words by combining the contexts of two semantically similar words to form a 'chimera'. we find that leap performs better than maml at adapting model parameters to a new corpus. we also conduct an extrinsic evaluation on ner in the biomedical domain where the results are comparable to maml, without improving in most cases on a random embedding baseline. finally, we examine which contexts are more beneficial to learn an embedding from, and note that the contexts from which an embedding is learned matters more than the meta-learning method employed. ## meta-learning meta-learning algorithms aim to capture knowledge across a variety of learning tasks such that fine-tuning a model on a specific task both avoids overfitting and leads to good performance. approaches include learning a similarity function by which to cluster and classify data points @xcite ; learning an update rule for neural network optimization @xcite ; and learning an initialization from which to fine-tune a model. we consider algorithms for the latter. formally, we consider task t to consist of a dataset d t of labelled examples (x, y) and loss function l t (θ) → r which maps a model's parameters θ to a real-valued loss. for batch gradient descent this loss is constant for a given set of parameters, however for stochastic gradient descent it depends on the sampled examples. during training, tasks t ∼ p(t ) are sampled from a distribution p(t ) over the tasks the model should be able to adapt to. the aim is then to train the model's parameters such that they capture features common to all tasks in p(t ), and thus are a promising initialization for any of these tasks. below we describe the approaches taken by maml and leap. ## leap for oov embedding learning in this section we describe our application of leap to learning embeddings for oov words. this technique is generally applicable to any word embed-ding regression function h θ trainable by gradient descent, though in our experiments we use the hice architecture @xcite . the regression function h θ is trained to predict a word's embedding given k contexts, and possibly morphological information, with words and their contexts sampled from some large training corpus d t . each word (either a training target or in a context) is represented as a pre-trained embedding in the input to h θ . we note that h θ can be trained only on words with sufficient occurrences in d t such that their pre-trained embeddings can be accurately learned. the trained h θ can then be used to infer the embeddings for oov words, for which we do not have a pre-trained embedding. however, oov words often form part of domain-specific vocabularies, the semantics of which are not captured by word embeddings trained on generic large corpora @xcite . to counteract this, we adapt the trained parameters θ t of the word embedding regression function to the domain in which we infer the oov embeddings. when adapting θ t we consider both d t , and the corpus on which we wish to infer oov embeddings, d n . we wish to transfer the knowledge encoded in θ t to the task of predicting oov embeddings for words in d n . one approach would be to simply fine-tune θ t on d n ; that is, sample words and their contexts from d n which have their pre-trained embeddings from d t known and train h θ as before on these words. this ignores that d n is much smaller than a corpus usually used for word embedding training, and direct training on it is likely to lead to overfitting to the corpus rather than adapting to its domain, which in turn hurts the quality of inferred oov embeddings, an instance of catastrophic forgetting @xcite . instead of fine-tuning, meta-learning algorithms can be applied. @xcite use maml, and we extend this work with the use of leap. we adapt θ t by applying leap across two tasks; inferring oov words in d t and d n . optimizing the objective in equation 4 moves the adapted parameters θ to minimize the lengths of the two corresponding learning processes. the loss function l(•|d) used throughout is the cosine distance between predicted and pre-trained embeddings. this pulls the model parameters θ along the learning processes for d t and d n originating from the pre-trained parameters θ t . the learning process for d t should already be short, as θ t is trained to convergence on d t . optimization of the pull-forward objective must naturally pull the parameters away from this point of convergence to minimize the length of the learning process for d n . however, as each task is weighted equally, the model parameters cannot move towards the convergence point for d n if this results in a larger divergence from the convergence point for d t . it is thus necessary for the parameters to move into areas which encode knowledge sharing between d t and d n . this reduces overfitting on d n by ensuring θ does not move too far from the convergence point for d t . while each word's embedding is inferred from only a few contexts, the total number of words available for training in d n is large enough so θ can be adapted over a larger number of gradient steps. we posit that this, in conjunction with knowledge sharing that considers the entire learning trajectory rather than just the beginning and end points, results in higher quality oov embeddings than those that can be obtained with maml. ## experiments 4.1 intrinsic evaluation to obtain an intrinsic evaluation of the methods proposed, we require a dataset that simulates the natural occurrences of oov words in a real-world setting, and defines a notion for evaluating how close an embedding is to representing an oov word's meaning. following @xcite , we use the 'chimera' dataset for evaluation, a popular benchmark dataset for oov words. the chimera dataset @xcite is constructed specifically to simulate unseen words occurring naturally in text. each unseen word is a chimera, which is a novel concept created by combining two related but distinct concepts; for example a gorilla and a bear. in total there are 33 chimeras, generated by first taking a base concept, called a pivot, and matching this with a compatible concept by traversing a list of terms ranked by similarity to the pivot. in the case of the 'gorilla/bear' chimera, the pivot is the gorilla and the compatible term is the bear. each chimera is then associated with passages of 2, 4 and 6 sentences, with half containing the pivot and half containing the compatible concept. the occurrences of the pivot and the compatible term are replaced with a nonce word that represents the chimera; for example 'mohalk'. each passage is then annotated by human subjects with similarity scores between the nonce word and six different words, called probes, specific to the chimera. these similarity scores are then averaged across human subjects, resulting in six scores for each passage indicating the similarity between the chimera and each probe word. for each of the 2-shot, 4-shot and 6-shot cases the chimera's embedding is inferred given each sentence (shot) as a context and the pivot's character sequence. following @xcite we measure the performance of the embeddings inferred by looking at their cosine similarity to the probe embeddings and calculating the spearman correlation to the similarity judgements by the human subjects. ## conclusion we investigated the use of meta-learning for fewshot learning of oov word embeddings. we built on the work of @xcite , formulating oov word embedding learning as a few-shot regression problem and training their proposed architecture hice to predict oov embeddings given k contexts and morphological features. we proposed the use of leap as a meta-learning algorithm to adapt hice to a new semantic domain and compared it to the popular maml as used by @xcite . experiments on a benchmark dataset show that leap is more stable and achieves comparably higher performance than maml in the context of oov embedding learning. further experimentation shows that there is little variation in which contexts perform well across both random seeds and meta-learning approaches, and a qualitative analysis indicates that performance is lower on ambiguous sentences with fewer content words. our findings suggest a future avenue of work which focuses on the selection of contexts from which to learn an oov embedding, such as prioritising contexts based on a notion of informativeness.