Unsupervised Domain Adaptation using Lexical Transformations and Label Injection for T witter Data (2023)

Abstract: Domain adaptation is an important and widely studied problem in natural language processing. A large body of literature tries to solve this problem by adapting models trained on the source domain to the target domain. In this paper, we instead solve this problem from a dataset perspective. We modify the source domain dataset with simple lexical transformations to reduce the domain shift between the source dataset distribution and the target dataset distribution. We find that models trained on the transformed source domain dataset performs significantly better than zero-shot models. Using our proposed transformations to convert standard English to tweets, we reach an unsupervised part-of-speech (POS) tagging accuracy of 92.14% (from 81.54% zero shot accuracy), which is only slightly below the supervised performance of 94.45%. We also use our proposed transformations to synthetically generate tweets and augment the Twitter dataset to achieve state-of-the-art performance for POS tagging.

## introduction in a typical machine learning setting, training, development and test sets are usually carved out of the same data collection effort. in doing this, we caveat our models with an implicit assumptionthe deployment dataset should belong to the same distribution as the training dataset. this is rarely the case and we see significant drops in performance when the model is deployed. the mismatch between the deployment data distribution, or target domain, and the training data distribution, or source domain, is known as domain shift @xcite and the process of adapting to target domain distributions is known as domain adaptation @xcite . the most widely studied domain adaptation methods are model-centric methods @xcite , where parts of the model, including the feature space, the loss function or even the structure of the model are altered @xcite @xcite . data-centric methods @xcite ) usually involve some form of bootstrapping and pseudo-labelling of the target domain data @xcite @xcite . a popular data-centric domain adaptation method is data selection, which is an intermediate training step that aims to select a subset of data that is closest to the target domain @xcite @xcite . we refer the reader to domain adaptation surveys in natural language processing for a detailed overview @xcite @xcite . to the best of our knowledge, none of the works we encounter in literature address the fundamental reason behind the need for domain adaptation -domain shift. if we are able to transform the source domain dataset such that the domain mismatch between the source domain and the target domain is reduced, while being able to exploit the annotations of the source domain corpus, then the models trained on such a transformed source domain data will naturally perform better on the target domain. this is the main motivation behind our work. all model-centric and data-centric domain adaptation methods can be applied on top of our proposed method and are complementary to it. in this paper, we transform the source domain dataset to resemble the target domain dataset more closely through a series of transformations. in our case, the source domain consists of standard english sentences and the target domain consists of tweets. through these transformations, we are able to improve the zero-shot pos tagging accuracy by 10.39% when averaged over five different bert models. also, when we combine the transformed data to augment the original target dataset, we achieve state-of-the-art pos tagging performance on the target dataset. ## lexical transformations and label injections standard english sentences and tweets have both semantic and lexical differences. tweets are more likely to be subjective and polarized (appendix a.1). on the other hand, tweets also contain unique lexical features like acronyms, emojis, user mentions, retweets, hashtags, as shown in figure 1 , and can be used as different parts of speech ( ## datasets in this paper, we work with two annotated pos tagging datasets. for standard english, we use the gum (georgetown university multilayer corpus) dataset @xcite . for twitter data, we use tweebank (tbv2) @xcite dataset. we choose these two datasets because they are both labelled using the universal dependencies @xcite framework, thus each of the datasets have identical 17 pos tags. the dataset statistics are shown in the gum dataset acts as our source domain dataset and is about 5 times larger than tbv2, which is our target domain dataset. gum dataset is made up of articles and interviews from wikinews, instructional articles from wikihow and travel guides from wikivoyage @xcite . the gum dataset contains longer sentences compared to the tweebank dataset. the tweebank dataset gets higher average polarity and subjectivity scores when compared to the gum dataset. the experiments analysing dataset properties are shared in appendix a.1. ## experiments in this section, we present four different types of lexical transformations and corresponding label injection methods for twitter as target domain. all transformations are performed on the gum train-split (the standard english dataset). models trained on the transformed dataset are tested on the tbv2 test set (the twitter dataset). all experiments shown in this paper report accuracy scores on tbv2 test set, in accordance with previous works @xcite @xcite . each experiment is repeated five times and the mean score is reported with standard deviations reported in brackets. all experiments in this paper are done using the huggingface implementations of different bert models. we use five different bert models, the original bert-base-uncased and bertlarge-uncased @xcite ) models, the roberta-base and roberta-large models @xcite and the bertweet model @xcite . ## results roberta-large and bertweet. the second section in ## conclusion a lot of focus in literature has been given to converting noisy social media text to standard english. in our work, we convert standard english into noisy social media-like text using simple lexical transformations and show that it can be used as an effective unsupervised domain adaptation and data augmentation method. the fundamental idea behind our work is to reduce domain shift by transforming the source domain into the target domain. we present experiments for these transformations between standard english and twitter domain and find an average accuracy boost for pos tagging of 10.39% across 5 different bert models, without ever using a single tweet for supervised training. department of j.p. morgan. j.p. morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy, or reliability of the information contained herein. this document is not intended as investment research or investment advice, or a recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product, or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person if such solicitation under such jurisdiction or to such person would be unlawful.