T rans POS : Transformers for Consolidating Different POS Tagset Datasets (2022)

Abstract: In hope of expanding training data, researchers often want to merge two or more datasets that are created using different labeling schemes. This paper considers two datasets that label part-of-speech (POS) tags under different tagging schemes and leverage the supervised labels of one dataset to help generate labels for the other dataset. This paper further discusses the theoretical difficulties of this approach and proposes a novel supervised architecture employing Transformers to tackle the problem of consolidating two completely disjoint datasets. The results diverge from initial expectations and discourage exploration into the use of disjoint labels to consolidate datasets with different labels.

## introduction there has been an explosion in the availability and variety of labeled datasets in almost every domain. unfortunately, artificial intelligence (ai) practitioners and researchers often find themselves unable to make use of labeled datasets for tasks related but not identical to their tasks. this is primarily due to different labeling schemes where a trivial mapping to merge the datasets into one larger dataset does not exist. in this paper, we explore the possibility of consolidating datasets that were curated for the same task with different labeling schemes. to make this easy to apply to any pair of datasets, we consider a very interesting scenario in which we attempt to make a model that can understand both datasets without ever actually seeing any examples that have labels from both of them. there are several domains and application areas to which our technique can be applied to, and frankly might be the only option. for example: when creating a data set to detect people, objects, and vehicles in an urban environment, we may want to supplement our existing data set with the popular cityscapes dataset @xcite , but struggle to directly apply those labels to the merged dataset due to a few minor differences in the label scheme, such as a smaller or larger label set. there could also be some information partially correlated with the existing dataset's labels; perhaps in our dataset we have to distinguish between standing and sitting people. cityscapes does not distinguish between these, so is it possible to use the label information (about where humans are) to get highquality segmentation under our desired labeling scheme? the focus of this paper comes from part-ofspeech (pos) tagging. although some tags are common to all datasets, different datasets may have different conventions for how to deal with more uncommon parts of speech, like modal verbs, particles, or even when to treat something as a noun. these problems are exacerbated in informal contexts. we provide a novel design for a supervised model that can translate labels from one dataset into another labeled dataset without requiring any shared examples. after analyzing results, we reconsider the situations under which it is possible to squeeze out extra performance from these labels, and show that it is unlikely for any kind of architecture to use label information to perform better than an equivalent model that does not, unless the architecture has access to shared examples or metadata about the meaning of the labels. ## setting let σ be the set of unicode characters. in our setting, we have two datasets that map from the space of sentences of unicode characters @xmath0. however, the two datasets use different labeling schemes: the first may use the standard universal pos tagset y while the second uses a proprietary pos tagset z. each sentence in the first dataset has a label for each word in presumably, y and z are very highly correlated, since they are both pos tags for a sentence, just defined with slightly different rules. we would like to expand the dataset d y to include the sentences and labels of d z , but unfortunately the labels are incompatible. however, we expect that we can still get useful information from the labels z. therefore, our goal is to build a predictor function f y : (x, z) → y that combines both the text and the information of the annotated d z to predict what the translated label would be in the tagset y. similarly, we consider the construction of f z : (x, y) → z. here are the two obvious baselines that could be used to construct f y : ## model in our method, we will transform our input x into an embedding space e using a transformer's encoder enc : @xmath1, one for each type of label y and z. d y : (e, z) → y and d z : (e, y) → z. then to infer a label z for a given training sample (x y , y), we can compute d z (enc(x y ), y). see figure 1 for a visualization. figure 1 : when evaluating the model, we use y and z as inputs! however, the setup used for validation will not work for training the model. ideally, we would like to make a loss function that penalizes the predicted value of z from being far from the true z corresponding to x y , but we do not have any access to the true z! we only have pairs (x, y) and (x, z), not (x, y, z). one way to fix this is to first predict ẑ using (x, y) and then use that as a surrogate for true z. to do this, we can create a supervised model s z that takes in the encoded variables enc(x y ) and outputs a prediction for the label z, which we can then input into the decoder d z . however, the careful reader will notice a flaw in this strategy: while d z takes the input as one-hot labels at inference time, it takes input as logits at training time. to reconcile this difference, we treat the softmax of the predicted logits as a probability distribution, from which we sample our true predicted label ẑ. the entire training process with the inputs d y is shown in figure 2 , and the model for d z is made the same way, but with the y / z inputs flipped. in our implementation, each mini-batch contains some examples from both datasets. to reduce the complexity of the model, we use the same base encoder model weights for s y , s z , d y , and d z , though in principal they could be different or only partially shared. the inquisitive reader may wonder why we use s z to predict labels z instead of reusing labels d z along with ground truth labels y. in this case, we will have given the label that we want to predict as an input to the model, and so the model can simply learn to predict the input! for example, suppose that in our model, rather than sampling ẑ ∼ s z (e y ), we reused the decoder weights to get ẑ := d z (e y , y), then predicted ỹ = d y (e y , ẑ) (and similarly on the other side). then, the model can simply learn to ignore the first argument of d y and d z and instead learn that d y (-, z) and d z (-, y) are inverses to each other. in this setup, it will perfectly predict all the training data, but it will be completely useless in practice. we actually tried this setup and found that the model would actually achieve performance competitive with the supervised model for a few epochs (perhaps due to regularization like dropout), but after training long enough, it learns the cheat and arrives at 0 training error and very high validation error. now, during training, y and z are completely derived from x. so, in principle, d y and d z may learn to ignore noisy outputs y and z and make predictions based solely on x. to prevent this, we enforce a very heavy dropout of 0.85 on the first term before passing it as input. the model and training code can be found in out github repository foot_0 in the footnote. ## datasets in this project, we consider two datasets: however; these two datasets have a data contamination problem: there are 210 identical shared tweets. in our case, however, this served as the perfect validation set for our model. looking at the distribution of the labels in this validation set (figure 3 ), we see that the ambiguity between the meanings of the labels will limit the performance of a direct mapping. the first baseline was created by making a 'direct map' between the labels. we looked at the validation set and chose the map that gave the highest possible score. the second 'supervised model' baseline was a normal transformer model; we train on the train split of one dataset and evaluate on the validation split of that same dataset. then, for 'our model', we trained with the described architecture, using the transformers bertweet (nguyen (2020)), and gpt-2 ( radford et al. (2019) ) a gpt-2 model and a bertweet model with the described architecture with dropout .85. after the training was complete, we evaluated the accuracy using the method described above to compute our model accuracy. to see if our model was really learning from the y labels, we used of the supervised model heads s y • enc and s z • enc to get the 'supervisor only' accuracy. this architecture is exactly the same as the baseline, but differences arise in the accuracy because the training process is not the same (in particular, there is very high x dropout). finally, we considered the accuracy of the full pipeline when there with 'no label input': instead of providing the z labels for the first dataset while predicting the y labels of the second, we just took the z labels that the model predicted and sampled from that distribution as we do at training time. ## conclusion the task of consolidating datasets with different labels and no shared examples is a hard problem. the experiments did not provide any improvement over the baseline of only using the x variables. this was surprising, as the correlation between the y and z labels is quite large. however, this may be due to an intrinsic difficulty with the setting (no shared examples) rather than the model design.