Treebank Embedding Vectors for Out-of-Domain Dependency Parsing (2020)

Abstract: A recent advance in monolingual dependency parsing is the idea of a treebank embedding vector, which allows all treebanks for a particular language to be used as training data while at the same time allowing the model to prefer training data from one treebank over others and to select the preferred treebank at test time. We build on this idea by 1) introducing a method to predict a treebank vector for sentences that do not come from a treebank used in training, and 2) exploring what happens when we move away from predefined treebank embedding vectors during test time and instead devise tailored interpolations. We show that 1) there are interpolated vectors that are superior to the predefined ones, and 2) treebank vectors can be predicted with sufficient accuracy, for nine out of ten test languages, to match the performance of an oracle approach that knows the most suitable predefined treebank embedding for the test set.

## introduction the universal dependencies project (nivre et al., 2016) has made available multiple treebanks for the same language annotated according to the same scheme, leading to a new wave of research which explores ways to use multiple treebanks in monolingual parsing (shi et al., 2017; sato et al., 2017; che et al., 2017; stymne et al., 2018). stymne et al. (2018) introduced a treebank embedding. a single model is trained on the concatenation of the available treebanks for a language, and the input vector for each training token includes the treebank embedding which encodes the treebank the token comes from. at test time, all input vectors in the test set of the same treebank are also assigned this treebank embedding vector. stymne et al. (2018) show that this approach is superior to mono-treebank training and to plain treebank concatenation. treebank embeddings perform at about the same level as training on multiple treebanks and tuning on one, but they argue that a treebank embedding approach is preferable since it results in just one model per language. what happens, however, when the input sentence does not come from a treebank? stymne et al. (2018) simulate this scenario with the parallel universal dependency (pud) test sets. they define the notion of a proxy treebank which is the treebank to be used for a treebank embedding when parsing sentences that do not come from any of the training treebanks. they empirically determine the best proxy treebank for each pud test set by testing with each treebank embedding. however, the question remains what to do with sentences for which no gold parse is available, and for which we do not know the best proxy. we investigate the problem of choosing treebank embedding vectors for new, possibly out-ofdomain, sentences. in doing so, we explore the usefulness of interpolated treebank vectors which are computed via a weighted combination of the predefined fixed ones. in experiments with czech, english and french, we establish that useful interpolated treebank vectors exist. we then develop a simple k-nn method based on sentence similarity to choose a treebank vector, either fixed or interpolated, for sentences or entire test sets, which, for 9 of our 10 test languages matches the performance of the best (oracle) proxy treebank. ## interpolated treebank vectors following recent work in neural dependency parsing (chen and manning, 2014; ballesteros et al., 2015; kiperwasser and goldberg, 2016; zeman et al., 2017, 2018), we represent an input token by concatenating various vectors. in our experiments, each word w i in a sentence @xmath0,...,w n ) is a concatenation of 1) a dynamically learned word vector, 2) a word vector obtained by passing the k i characters of w i through a bilstm and 3), following stymne et al. (2018), a treebank embedding to distinguish the m training treebanks: ## data and resources for all experiments, we use ud v2.3 (nivre et al., 2018). we choose czech, english and french as our development languages because they each have four treebanks (excluding pud), allowing us to train on three treebanks and test on a fourth. for testing, we use the pud test sets for languages for which there are at least two other treebanks with training data: czech, english, finnish, french, italian, korean, portuguese, russian, spanish and swedish. following stymne et al. (2018 ), we use the transition-based parser of de lhoneux et al. (2017) with the token input representations as eq. 1 above. source code of our modified parser and helper scripts to carry out the experiments are available online. foot_0 4 are interpolated treebank vectors useful? we attempt to ascertain how useful interpolated treebank embedding vectors are by examining the labelled attachment score (las) of trees parsed with different interpolated treebank vectors. for each of our three development languages, we train multi-treebank parsing models on the four combinations of three of the four available treebanks and we test each model on the development sets of all four treebanks, i. e. three in-domain parsing settings and one out-of-domain setting. @xcite since @xmath1, all treebank vectors lie in a plane and we can visualise las results in colour plots. as the treebank vectors can have arbitrary distances, we plot (and sample) in the weight space r m . we include the equilateral triangle spanned by the three fixed treebank embedding vectors in our plots. points outside the triangle can be reached by allowing negative weights α @xmath2. we obtain treebank las and sentence-level las for 200 weight vectors sampled from the weight space, including the corners of the triangle, and repeat with different seeds for parameter initialisation and training data shuffling. rather than sampling at random, points are chosen so that they are somewhat symmetrical and evenly distributed. figure 1 shows the development set las on cs cac-dev for a model trained on cs cltt+fictree+pdt with the second seed. we create 432 such plots for nine seeds, four training configurations, four development sets and three languages. the patterns vary with each seed and configuration. the smallest las range within a plot is 87.8 to 88.3 (cs cac+cltt+pdt on cs pdt with the seventh seed). the biggest las range is 59.7 to 76.8 (fr gsd+sequoia+spoken on fr spoken with the fifth seed). the location of the fixed treebank vectors e 3 (t) are at the corners of the triangle in each graph. for in-domain settings one or two corners usually have las close to the highest las in the plot. the best las scores (black circles), however, are often located outside the triangle, i. e. negative weights are needed to reach it. turning to sentence-level las, figure 2 shows the las for an individual example sentence rather than an entire development set. this sentence is taken from en partut-dev and is parsed with a model trained on en ewt+gum+lines. for this 28-token sentence, las can only change in steps of 1/28 and 34 of the 200 treebank embedding weight points share the top score. negative weights are needed to reach these points outside the triangle. over all development sentences and parsing models, an interpolated treebank vector achieves highest las for 99.99% of sentences: in 78.07% of cases, one of the corner vectors also achieves the highest las and in the remaining 21.92%, interpolated vectors are needed. it is also worth noting that, for 39% of sentences, las does not depend on the treebank vectors at all, at least not in the weight range explored. often, las changes from one side to another side of the graph. the borders have different orientation and sharpness. the fraction of points with highest las varies from few to many. the same is true for the fraction of points with lowest las. noise seems to be low. most data points match the performance of their neighbours, i. e. the scores are not sensitive to small changes of the treebank weights, suggesting that the observed differences are not just random numerical effects. this preliminary analysis suggests that useful interpolated treebank vectors do exist. our next step is to try to predict them. in all subsequent experiments, we focus on the out-of-domain setting, i. e. each multi-treebank model is tested on a treebank not included in training. ## predicting treebank vectors we use k-nearest neighbour (k-nn) classification to predict treebank embedding vectors for an individual sentence or a set of sentences at test time. we experiment with 1) allocating the treebank vector for an input sentence using the k most similar training sentences (se-se), and 2) allocating the treebank vector for a set of input sentences using the most similar training treebank (tr-tr). we will first explain the se-se case. for each input sentence, we retrieve from the training data the k most similar sentences and then identify the treebank vectors from the candidate samples that have the highest las. to compute similarity, we represent sentences either as tf-idf vectors computed over character n-grams, or as vectors produced by max-pooling over a sentence's elmo vectors (peters et al., 2018) produced by averaging all elmo bilm layers. @xcite we experiment with @xmath3, 3, 9. for many sentences, several treebank vectors yield the optimal las for the most similar retrieved sentence(s), and so we try several tie-breaking strategies, including choosing the vector closest to the uniform weight vector (i. e. each of the three treebanks is equally weighted), re-ranking the list of vectors in the tie according to the las of the next most similar sentence, and using the average las of the k sentences retrieved to choose the treebank vector. three treebank vector sample sizes were tried: we include oracle versions of each k-nn model in our experiments. the k-nn oracle method is different from the normal k-nn method in that the test data is added to the training data so that the test data itself will be retrieved. this means that a k-nn oracle with @xmath4. in the tr-tr setting, our k-nn classifier is selecting one of three treebanks for the fourth test treebank. in the oracle k-nn setting, it selects the test treebank itself and parses the sentences in that treebank with its best-performing treebank vector. when the treebank vector sample space is limited to the vectors for the three training treebanks (fixed), this method is the same as the best-proxy method of stymne et al. (2018). ## results the development results, averaged over the four development sets for each language, are shown in tables 1 and 2 . @xcite as discussed above, upper bounds for k-nn prediction are calculated by including an oracle setting in which the query item is added to the set of items to be retrieved, and k restricted to 1. we are also curious to see what happens when an equal combination of the three fixed vectors (uniform weight vector) is used (equal), and when treebank vectors are selected at random. the oracle k-nn results are shown in the third section of our equal method, that uses the weights ( 1 ⁄3, 1 ⁄3, 1 ⁄3), is shown in the last row of ## conclusion in experiments with czech, english and french, we investigated treebank embedding vectors, exploring the ideas of interpolated vectors and vector weight prediction. our attempts to predict good vector weights using a simple regression model yielded encouraging results. testing on pud languages, we match the performance of using the best fixed treebank embedding vector in nine of ten cases within the bounds of statistical significance and in five cases exactly match it. on the whole, it seems that our predictor is not yet good enough to find interpolated treebank vectors that are clearly superior to the basic, fixed vectors and that we know to exist from the oracle runs. still, we think it is encouraging that performance did not drop substantially when the set of candidate vectors was widened (α @xmath6. we do not think the superior treebank vectors found by the oracle runs are simply noise, i. e. model fluctuations due to varied inputs, because the las landscape in the weight vector space is not noisy. for individual sentences, las is usually constant in large areas and there are clear, sharp steps to the next las level. therefore, we think that there is room for improvement for the predictor to find interpolated vectors which are better than the fixed ones. we plan to explore other methods to predict treebank vectors, e. g. neural sequence modelling, and to apply our ideas to the related task of language embedding prediction for zero-shot learning. another area for future work is to explore what information treebank vectors encode. the previous work on the use of treebank vectors in mono-and multi-lingual parsing suggests that treebank vectors encode information that enables the parser to select treebank-specific information where needed while also taking advantage of treebank-independent information available in the training data. the type of information will depend on the selection of treebanks, e. g. in a polyglot setting the vector may simply encode the language, and in a monolingual setting such as ours it may encode annotation or domain differences between the treebanks. interpolating treebank vectors adds a layer of opacity, and, in future work, it would be interesting to carry out experiments with synthetic data, e. g. varying the number of unknown words, to get a better understanding of what they may be capturing. future work should also test even simpler strategies which do not use the las of previous parses to gauge the best treebank vector, e. g. always picking the largest treebank.