Bidirectional Domain Adaptation Using Weighted Multi-Task Learning (2021)

Abstract: Domain adaption in syntactic parsing is still a significant challenge. We address the issue of data imbalance between the in-domain and out-of-domain treebank typically used for the problem. We define domain adaptation as a Multi-task learning (MTL) problem, which allows us to train two parsers, one for each do-main. Our results show that the MTL approach is beneficial for the smaller treebank. For the larger treebank, we need to use loss weighting in order to avoid a decrease in performance be-low the single task. In order to determine towhat degree the data imbalance between two domains and the domain differences affect results, we also carry out an experiment with two imbalanced in-domain treebanks and show that loss weighting also improves performance in an in-domain setting. Given loss weighting in MTL, we can improve results for both parsers.

## introduction domain adaption in syntactic parsing is still a significant challenge. while recent work has shown steady improvements, we have not necessarily achieved proportionally better results as expected with neural models @xcite . one simple reason can be attributed to the fact that we have severe limitations in terms of existing data, data sizes, and the imbalance between the two treebanks typically present in domain adaptation settings. multi-task learning (mtl; @xcite allows for joint learning, which can help facilitate cross information sharing between tasks. this has proven particularly beneficial for tasks that have large data imbalances, with the smaller data tasks benefiting substantially more @xcite @xcite , and should thus also be useful in domain adaptation. we define domain adaptation as an mtl problem where the two tasks correspond to training on two treebanks from different domains. note that in this setting, we do not have a primary and a secondary task, but instead we can interpret both tasks as primary. one of the inherent difficulties we face in crossdomain parsing are large discrepancies in the size of the treebanks. this creates a default training scenario of imbalance across the domains that should benefit the smaller domain, but it may inversely impact the larger domain, resulting in a degradation in performance due to negative transfer in a multi-task learning model, compared to their single task (stl) baselines. we investigate here whether it is possible to control the negative impact of the smaller domain on the larger one, and how the imbalance factor impacts this balance. this means that in all cases, we evaluate on both domains. more specifically, we investigate the following questions: neural networks have increased the ability and ease by which models can exploit information sharing. much recent parsing research has examined the impact of parameter sharing across treebanks and languages @xcite , though often not explicitly within an mtl setup, where different treebanks/languages are treated as multiple tasks. soft sharing of parameters has proven effective on treebanks of the same annotation style in lower resource settings @xcite as well as for multiple treebanks of the same language when hard sharing all other parameters @xcite . however, sharing too many parameters between unrelated languages has been shown not to be beneficial @xcite . more explicit mtl settings with treebanks representing different individual tasks have proven successful in array of settings across languages and architectures @xcite @xcite . ## stl vs. mtl learning curves our first set of experiments concerns the question how different levels of size imbalance in the training data from the two domains affect an mtl parser. figure 2 presents learning curves comparing the german treebanks, figures 3 and 4 compare the two italian twitter treebanks to isdt. across all settings, we see that using only word embeddings gives the lowest results, followed by word plus pos tag embeddings, with word plus pos tag plus bert embeddings resulting in the highest las. also across all experiments, we see that sharing the mlp layers (in green) is slightly more beneficial to the smaller twitter treebanks than having separate layers (in blue). the differences between the shared and unshared model are even smaller for the large treebanks. the two german treebanks have very similar performance at 1k while there is a considerable difference between the italian trebanks, with the twitter treebanks being more difficult to parse than isdt. when looking at the results on german in figure 2 , we see that in all settings, the mtl setup is beneficial for the tweede treebank, independent of the amount of out of domain training data. the same holds initially for the gsd treebank. however, as more gsd data becomes available, the mtl setup starts to be detrimental for the gsd parser, and the results stay below the stl baseline. additionally, the type of embeddings plays a factor in the overall improvement: while the curves flatten out quickly for word+pos and word+pos+bert embeddings, the curve for word embeddings indicates that adding the full set still results in gains. it is also concerning that for the word+pos and word+pos+bert embeddings, the results on gsd start decreasing after 4k gsd sentences. we attribute this to negative transfer, i.e., the mtl setting is more focused on finding an optimal solution between the non-twitter and the twitter task, resulting in a degradation in performance of the non-twitter task from the equally prioritized twitter signals. note that while this makes sense from a mtl point of view, it is counter-intuitive from a domain adaptation perspective: for the gsd task, this means that adding more in-domain data results in lower in-domain performance. the results for the italian treebanks in figures 3 and 4 exhibit similar, but not identical trends to the german experiments. one noticeable difference is that the isdt improvements tend to be far more linear, where we only begin to see a flattening of the curve at the full training size. another more interesting difference concerns the point where the stl on the large treebank (gsd or isdt) improves over the mtl model: for german, this happens systematically at 4k, for italian, it ranges from 4-8k for twittir ò and 2-8k for postwita depending on the type of embeddings. we see that for word+pos+bert, the stl is already on par with the mtl model at 2k, but the word embeddings model needs closer to 8k. ## loss weighting we now turn to the question of loss weighing, i.e., can we address the data imbalance problem if we assign higher weights to one or the other task in the loss function? using loss weighting would give us a principled way of handling the data imbalance. for a description of the weighting scheme, see section 3.3. for the experiments in this section, all mtl tasks are learned using the shared mlp setting (as it performed better in most set-tings of section 4) and word+pos embeddings (as it performed better than word-only embeddings but does not require pretraining on external sources, such as bert), and we vary the training size of the large treebank between 1k, 2k, 4k, and all available training data. figure 5 shows the results for the german treebanks, figure 6 shows the combination of the italian isdt and twittir ò, and figure 7 the combination of isdt and postwita. each graph shows the results for the two tasks per setting. all experiments in section 4 are based on standard alternating loss for both tasks, i.e., the loss for each task is used as is, which is equivalent to having a weight of 1. for the experiments here, the x-axis denotes the weights applied to the large treebank task, ranging from 0 to 1. the corresponding weight applied to the second task is 1 -x, i.e., it decreases from 1 to 0. for each weight, we report the las for either task, blue denoting the large treebank and orange the twitter treebank. the vertical dotted lines mark the 0.5 weight setting, where both tasks are weighted equally, and the horizontal dotted lines denote the performance of the mtl parser in this setting. the diamond markers in each line denote the best performance achieved by weighted mtl parser for that task. looking at the german results in figure 5 , we see that in the 1k setting, the optimal performance for gsd is reached when each task is assigned a weight of 0.5. for tweede, the optimal performance is reached in the same area, namely for a weight setting of 0.45:0.55 for gsd:tweede. in the 2k setting, even though the optimal performance shifts towards higher weights for the large treebank (0.85 for gsd and 0.4 for tweede), the actual increase in performance is minimal compared to the balanced weight setting. this trend continues as we add more data to gsd (for 4k and full settings): the optimal weights move closer to the maximal weight for the large treebank. this shows that as the treebank sizes become more imbalanced, the optimal performance is reached by applying a higher weight to the large treebank and consequently a lower weight to the small twitter treebank. it is also interesting to see that for the 1k setting, the tweede results surpass the gsd results. this suggests that the tweede data are syntactically easier than the gsd ones. we also see that the performance for tweede starts plateauing once it reaches the 0.5 weight setting for imbalanced scenarios (2k, 4k and full). hence, for tweede, the performance gain in the optimal setting beyond 0.5 is smaller than for gsd. moreover, while the mtl models using balanced weights (section 4) do not always improve over the stl baselines, weighted mtl improves in all settings and all tasks. for gsd, the highest gain over the stl occurs in the 1k setting. the more data we add, the smaller the improvement. for the full gsd set, the improvement is minimal. the results for the two italian experiments, in figure 6 for the combination of isdt and twit-tir ò, and in figure 7 for the combination of isdt and postwita, show similar trends with some differences. for both experiments, the twitter task shows slightly concave curves for 1k and 2k respectively. starting from 4k, we see an upward trend for isdt as the weight for the large task increases. for german, this only occurs with the full gsd training set. these results show that for smaller data sizes, i.e., when the two treebanks are closer to balanced, weighting does not matter that much: for most settings, the optimal results are close to a 0.5 weight. as the data imbalance increases, it becomes more important to slide the weights towards the larger task in order to avoid negative transfer. the weighted mtl models allow us to use the maximum data for the larger treebank and even reach a slight gain in performance over the stl. the twitter treebanks show even more improvement than in the standard mtl setup, suggesting that they benefit from a reduction of their own signal along with a more powerful signal from the large treebank. weighting the different losses is akin to having different learning rates for each task (i.e., treebank), impacting the contribution each has on the step sizes derived from the loss in finding an optimal solution @xcite . when the data sizes of the two domains are closer, the steps taken by each benefit each other. as the non-twitter data increases however, the twitter data have too much influence on the step sizes, resulting in a degradation on the non-twitter treebank since standard mtl optimizes both equally. while mtl is thought to help overcome local optima that can occur in an stl @xcite , in our case, we assume that the weights have a similar effect: they help both the non-twitter and twitter models overcome local optima encountered in a standard mtl setup. ## best results in ## conclusion & future work we have investigated the use of mtl for domain adaptation in parsing to address the data imbalance. the effectiveness of mtl depends on many interacting factors as laid out in section 2.3. important factors we directly examine in our experiments are data size imbalances, difficulty of tasks, and task learning rates. our learning curves for german and italian show that mtl underperforms an stl when the data size imbalances become too great, due to negative transfer in optimizing for two tasks. additionally, both our out-of-domain and in-domain experiments demonstrate that task learning difficulty affects both setups, even with balanced data. by using loss weighting, we are able to influence the learning rates of individual tasks, which helps reduce the negative transfer caused by both the data size imbalances and tasks difficulties. this allows us to train weighted mtl models where both parsers are able to outperform both stl and non-weighted mtl models on both treebanks at the same time, even when highly imbalanced, for in-domain and out-of-domain experiments for both german and italian. we conclude that while domain differences certainly play a factor, data imbalance appears to have more influence on parser performance. in the future, our experiments need to be extended to a wider range of languages and target domains. additionally, we will investigate strategies for dynamic learning of weights @xcite @xcite for determining optimal loss weighting automatically, as well as more complex scheduling approaches @xcite @xcite to further improve performance.