Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks (2020)

Abstract: Multi-task learning (MTL) and transfer learning (TL) are techniques to overcome the issue of data scarcity when training state-of-the-art neural networks. However, finding beneficial auxiliary datasets for MTL or TL is a time- and resource-consuming trial-and-error approach. We propose new methods to automatically assess the similarity of sequence tagging datasets to identify beneficial auxiliary data for MTL or TL setups. Our methods can compute the similarity between any two sequence tagging datasets, they do not need to be annotated with the same tagset or multiple labels in parallel. Additionally, our methods take tokens and their labels into account, which is more robust than only using either of them as an information source, as conducted in prior work. We empirically show that our similarity measures correlate with the change in test score of neural networks that use the auxiliary dataset for MTL to increase the main task performance. We provide an efficient, open-source implementation.

## introduction state-of-the-art neural networks usually require large amounts of training data and vast computational resources. especially for low-resource tasks, data scarcity is the main issue hampering the training of robust models. by leveraging multitask learning or transfer learning, auxiliary data can be incorporated into the training to boost the main task performance. finding suitable auxiliary datasets for these cases is a time-and resourceconsuming trial-and-error approach, because there can be plenty of plausible auxiliary datasets that could help to learn the main task. for a proper evaluation of different auxiliary datasets, hyperparameter search and training runs with multiple random seeds have to be performed for each auxiliary dataset individually. thus, the process takes even longer and uses even more computational resources. we propose methods to shorten this trial-and-error approach by computing the similarity between any two sequence tagging datasets. based on the similarity, suitable datasets can be quickly selected to be used as auxiliary training data for multi-task or transfer learning. our contributions are a family of novel methods to compute the similarity of sequence tagging datasets, where the similarity values correlate with the change in multi-task learning performance when using one dataset as auxiliary data for training the other. we evaluate our methods in experiments with five part-of-speech (pos) tagging, nine named-entity recognition (ner) and three argumentation mining (am) datasets. our similarity measures allow for comparison both datasets for the same and different tasks, not requiring the same set of labels on target and auxiliary dataset. the calculated similarity scores can be used to predict which dataset will be beneficial as auxiliary training data for multi-task training in order to shorten the search process. ## information-theoretic clustering comparison measures entropy is a measure of the uncertainty of a random variable. the entropy h(x) of a discrete random variable x with alphabet x is defined as joint entropy h(x, y ) extends entropy from a single to two random variables. for a pair of discrete random variables (x, y ) with a joint probability distribution p(x, y), it is defined as mutual information (mi) i(x; y ) describes the amount of information one random variable x contains about another y . it is a symmetric measure of range [0, min{h(x), h(y )}] defined as with probability mass functions p(x), p(y) and a joint probability mass function p(x, y). for a detailed description of entropy, mutual information and information theory in general, please refer to @xcite . a clustering c is a way to partition a dataset d into non-overlapping subsets {c 1 , c 2 , . . . } together containing all n items of d. comparing clusterings requires a measure to determine the quality of a clustering according to another clustering, e.g. the ground truth. such a measure should quantify the amount of information shared between both clusterings. @xcite information-theoretic clustering comparison measures are based on a solid mathematical foundation from information theory and can work with non-linear similarities. they have become popular by the works of @xcite and @xcite . mutual information measures the information shared between two clusterings c and c . a higher mi signals a greater help in predicting the cluster labels in c with information from c . several normalized mutual information variants can be derived: analogously to n m i max , there are n m i sum , n m i sqrt and n m i min that use entropy sums, square root of the entropy products or minimum of both entropy values as a normalization factor @xcite @xcite . they are all bounded in [0, 1], equaling 0 when two clusterings share no information at all, i.e. are fully independent and 1 when two clusterings are identical. according to @xcite , n m i max and n m i joint satisfy the highest number of theoretical properties desirable among the clustering comparison measures. they prove that only the unit complements of both measures satisfy the metric property (positive definiteness, symmetry and triangle inequality). while all measures satisfy the normalization property, none conform to the constant baseline property unless the number of items n is large, compared to the number of clusters. ## method the high-level idea of our dataset similarity measures is the following: words and labels from one dataset are correlated with the words and their labels from another dataset to create a probabilistic mapping between both label sets. either an exact string matching or a fuzzy matching based on word embedding representations can be used. the dataset similarity is measured via the quality of this label mapping. ## experiments in this section, experiments will be performed to check whether the similarity of two datasets correlates with the effect on the mtl performance when using the second dataset as auxiliary training data. ## conclusion the similarity measures allow distinguishing good from bad candidates for usage as auxiliary data. this is an immensely valuable information as the number of expensive neural network training runs can be reduced to a fraction while still finding the best auxiliary dataset(s) to increase performance on the main task. in contrast to previous methods, our measures do not require the label sets to be the same and do not require automatic tagging. the experiments show that similarity measures allow ordering the effects of auxiliary datasets by direction and intensity for an individual training dataset. our experimental findings are also supported from a theoretical point of view. the developed methods working on both words and their labels have a substantial advantage over approaches that are based only on words or the label distributions. the quick similarity calculation can improve the main task performance when better datasets are used as auxiliary data that would never have made it through the otherwise purely manual preselection process. in future work, apart from improving the similarity measures, it could be examined to predict mtl scores or estimate the right amount of auxiliary data or shared parameters in the neural network.