Using dependency parsing for few-shot learning in distributional semantics (2022)

Abstract: In this work, we explore the novel idea of employing dependency parsing information in the context of few-shot learning, the task of learning the meaning of a rare word based on a limited amount of context sentences. Firstly, we use dependency-based word embedding models as background spaces for few-shot learning. Secondly, we introduce two few-shot learning methods which enhance the additive baseline model by using dependencies.

## introduction distributional semantics models create word embeddings based on the assumption that the meaning of a word is defined by the contexts it is used in (for an overview, see @xcite @xcite . a fundamental challenge for these approaches is the difficulty of producing high-quality embeddings for rare words, since the models often require vast amounts of training examples @xcite . to address this problem, various few-shot learning methods have been previously introduced. the goal of a few-shot learning technique is to learn an embedding that captures the meaning of a word, given only a few context sentences. the rare word's vector has to be placed in an existing background space of embeddings. few-shot learning in distributional semantics is a relatively underexplored area, with important practical applications. having good representations of rare words is highly desirable in applications aiming to understand dialects or regionalisms, as well as specific technical language. in this work, we explore the idea of incorporating information from the dependency parse of sentences in the context of few shot-learning. an intuition why this might be useful is provided in figure 1 . in the given sentence, the most relevant word for inferring the meaning of the target rare word "conflagration" is "destroyed". even if this word is located far from the target, it is directly connected to it through a nominal subject dependency. moreover, the fact that the target word is used in a certain dependency structure might reveal important characteristics related to its meaning. since in the case of few-shot learning the data is limited, using dependency parsing information is a resource with great potential to boost existing models. as a first effort in this direction, this work provides three contributons. firstly, we explore the effect of using dependency-based word embeddings as background spaces. secondly, we introduce new few-shot learning methods leveraging the dependency parsing information. lastly, we update a previous dependency-based background model to make it more suitable for few-shot learning. ## background: dependency-based word embeddings the widely-used skip-gram model introduced by @xcite takes the contexts of a word to be those words surrounding it in a pre-defined window size. the model learns the embeddings in an unsupervised manner, using a feed-forward neural network trained on large amounts of sentences. @xcite proposed a different way to construct the contexts of a target word in the training process of the skip-gram model. instead of taking the words from a pre-defined window, one takes the words that are connected to the target word by a syntactic dependency. the contexts were defined as the concatenation of the connected word and the label of the dependency. this allowed the model to differentiate between same words used in different syntactic roles. the dependency-based word embeddings were found to be better at capturing similarity, while the window-based models capture relatedness. for example, a dependency-based model would produce close embeddings for "rome" and "florence", which are syntactically similar since they can be the severe conflagration which affected troy on 10 may 1862 destroyed 507 buildings. amod det relcl nsubj figure 1 : a dependency parse, illustrating that context words connected by a dependency can be important for inferring meaning in a few-shot setting, such as "destroyed" for the rare word "conflagration". used in the same grammatical contexts, while a window-based model is likely to place closely the embedding of highly related terms such as "rome" and "ancient", even if they cannot be used interchangeably since they are different parts of speech. levy and goldberg's model successfully captured syntactic similarity, but failed to express how different dependency types affect relations between words. moreover, it introduced sparsity issues. @xcite developed the dependency matrix model to address these shortcomings. instead of incorporating the dependency labels in the context vocabulary, each dependency type d is associated with a matrix t d , which acts as a meaning representation of the link between the target and the context words. the matrices t d , as well as the vectors holding the target vectors e and context vectors o, are learned during training. let d be the set of training examples given by tuples of target word t, context word c and dependency type d. for each tuple, we generate a set d of negative samples (t, c , d) by drawing context words c from a noise distribution and maintaining the same target word t and dependency type d. the learning goal is to maximise the function in (1), where σ is the sigmoid function and e t and o c are the vectors of the target and context word. as a straight-forward yet successful baseline, the vector of the rare word is estimated by the sum of the vectors of the words in contexts, as proposed by @xcite and @xcite . the latter noticed that not including the stop-words greatly improves the performance on the evaluation tasks. to optimise the performance of the additive model, @xcite proposed weighting the context words according to distance and frequency, as well as subtracting a "negative sampling" vector. these modifications take hyperparameters that are important for skip-gram's strong performance, such as number of negative samples k and window size n @xcite , and apply them to the few-shot setting. for each word w in the vocabulary v , with frequency f (w) and distance m from the target rare word t, and for a frequency threshold τ , we calculate the subsampling weight s(w), the window weight r(w) and negative sampling coefficient n(w). assume c is the collection of non-stop context words for the given target rare word t and v c is the vector in the background space for each @xmath0. the vector of the target rare word t will is: more involved models have been proposed for the task of few-shot learning. @xcite introduced a la carte, which applies a linear transformation to the sum of the context words obtained by the additive model. the weights of the linear transformation are optimised based on the cooccurrence matrix of the corpus. @xcite takes this approach further in the neural a la carte model, by using a neural network with a hidden layer to produce a non-linear transformation matrix, which adds flexibility. the meaning of a rare word can often be deduced from the word form itself. this information has been leveraged in few-shot learning models. for example, the form-context model @xcite is a hybrid method which retrieves the weighted sum between the surface form embedding of the rare word, obtained using fasttext @xcite and the context-based embedding, produced using the a la carte model. in this paper, we focus on additive methods, which do not require additional training on fewshot learning examples. this keeps the inference fast and in line with the true few-shot learning setting proposed by @xcite . ## dependency-based fsl methods dependency relations proved to be an informative tool in the context of creating distributional semantics models. based on this success, we introduce two dependency-based few-shot learning methods which build on top of the additive model. in this section, we assume we have already trained a background space of embeddings v i for each word i. in our setup, we chose to consider only the target embeddings learnt by the aforementioned background models, i.e. v @xmath1. alternatively, one could use the concatenation of the target and context embeddings. ## experiments in our setup, we considered three background models: window-based skip-gram, dependency-based skip-gram and the modified dependency matrix model which only uses one embedding for each word. to allow a direct comparison, we trained them all on the wikiwoods @xcite snapshot of english wikipedia. the same hyperparameters were used: a dimensionality of 100, 15 negative samples, a batch size of 5, and an adagrad optimiser with an initial learning rate of 0.025. for the dependency models, we used the universal dependency parser provided by spacy @xcite . we applied the two few-shot methods we devised, as well as the additive model with window weighting, subsampling and negative sampling described in §3. the hyperparameters were @xmath2, @xmath3. ## conclusion we investigated the use of dependency information for few-shot learning in distributional semantics. we found that dependency-based contexts are more useful than window-based contexts, with better performance across three evaluation datasets. we proposed a simplified version of the dependency matrix model, using only one vector per word, which makes it easier to apply in a few-shot setting. an important next step would be to investigate the use of the proposed methods for other languages, since our work was limited to english data and it is possible that the dependency structure is more relevant for few-shot learning in the case of specific languages. in order to do such an analysis, one would additionally need to create test data for the few shot-learning tasks, which would require the participation of speakers of the selected languages. in future work, performance might be further improved by training an a la carte model (discussed in §3), where the use of dependencies would make it possible to use a graph-convolutional network @xcite .