  Word embedding learning methods require a large number of occurrences of a
word to accurately learn its embedding. However, out-of-vocabulary (OOV) words
which do not appear in the training corpus emerge frequently in the smaller
downstream data. Recent work formulated OOV embedding learning as a few-shot
regression problem and demonstrated that meta-learning can improve results
obtained. However, the algorithm used, model-agnostic meta-learning (MAML) is
known to be unstable and perform worse when a large number of gradient steps
are used for parameter updates. In this work, we propose the use of Leap, a
meta-learning algorithm which leverages the entire trajectory of the learning
process instead of just the beginning and the end points, and thus ameliorates
these two issues. In our experiments on a benchmark OOV embedding learning
dataset and in an extrinsic evaluation, Leap performs comparably or better than
MAML. We go on to examine which contexts are most beneficial to learn an OOV
embedding from, and propose that the choice of contexts may matter more than
the meta-learning employed.
