DILBERT : Customized Pre-Training for Domain Adaptation with Category Shift, with an Application to Aspect Extraction (2021)

Abstract: The rise of pre-trained language models has yielded substantial progress in the vast majority of Natural Language Processing (NLP) tasks. However, a generic approach towards the pre-training procedure can naturally be sub-optimal in some cases. Particularly, fine-tuning a pre-trained language model on a source domain and then applying it to a different target domain, results in a sharp performance decline of the eventual classifier for many source-target domain pairs. Moreover, in some NLP tasks, the output categories substantially differ between domains, making adaptation even more challenging. This, for example, happens in the task of aspect extraction, where the aspects of interest of reviews of, e.g., restaurants or electronic devices may be very different. This paper presents a new fine-tuning scheme for BERT, which aims to address the above challenges. We name this scheme DILBERT: Domain Invariant Learning with BERT, and customize it for aspect extraction in the unsupervised domain adaptation setting. DILBERT harnesses the categorical information of both the source and the target domains to guide the pre-training process towards a more domain and category invariant representation, thus closing the gap between the domains. We show that DILBERT yields substantial improvements over state-of-the-art baselines while using a fraction of the unlabeled data, particularly in more challenging domain adaptation setups.

## introduction aspect-based sentiment analysis (absa) @xcite , extracting aspect-sentiment pairs for products or services from reviews, is a widely researched task in both academia and industry. absa allows a fine-grained and realistic evaluation of reviews, as real-world reviews typically 1 our code is publicly available at: https://github.com/tonylekhtman/dilbert figure 1 : a review from the restaurants domain. we use bio tags to mark the spans of the aspects. each aspect belongs to one category: for example, the waiter aspect belongs to the service category. in this review, the word food serves as both an aspect and a category. in this paper, we focus on the sub-task of aspect extraction (ae, a.k.a opinion targets extraction): extracting from opinionated texts the aspects on which the reader conveys sentiment. for example, in figure 1 , the waiter, food, and the views of the city are aspects derived from broader categories: service, food, and location. this task is characterized by a multiplicity of domains, as reviews and other opinionated texts can be written about a variety of products, services as well as many other issues. moreover, the aspect categories of interest often differ between these domains. as for most nlp tasks and applications, ae research has recently made substantial progress. while transformer @xcite based pre-trained models @xcite have pushed results substantially forward, they rely on in-domain labeled data to achieve their strong results. annotating such data for multiple domains is costly and laborious, which is one of the major bottlenecks for developing and deploying nlp systems. as noted above, ae forms a particu-larly challenging variant of the domain adaptation problem, as the aspect categories of interest tend to change across domains. a well-established approach for addressing the above bottleneck is domain adaptation (da) @xcite . da, training models on source domain labeled data so that they can effectively generalize to different target domains, is a long-standing research challenge. while the target domain labeled data availability in da setups ranges from little (supervised da @xcite ) to none (unsupervised da @xcite ), unlabeled data is typically available in both source and target domains. this paper focuses on unsupervised da as we believe it is a more realistic and practical scenario. due to the great success of deep learning models, da through representation learning @xcite , i.e., learning a shared representation for both the source and target domains, has recently become prominent @xcite . of particular importance to this line of work are approaches that utilize pivot features @xcite that: (a) frequently appear in both domains; and (b) have high mutual information (mi) with the task label. while pivot-based methods achieve state-of-the-art results in many text classification tasks @xcite @xcite , it is not trivial to successfully apply them on tasks such as ae. this stems from two reasons: first, ae is a sequence tagging task with multiple labels for each input example (i.e., word-level labels for input sentences). for a feature to meet the second condition for being a pivot (high mi with the task label), further refinement of the pivot definition is required. second, different domains often differ in their aspect categories and hence if a feature is highly correlated with a source domain label (aspect category), this is not indicative of its being correlated with the (different) aspect categories of the target domain. to overcome these limitations, we present dil-bert: domain invariant learning with bert, a customized fine-tuning procedure for ae in an unsupervised da setup. more specifically, dilbert employs a variant of the bert masked language modeling (mlm) task such that hidden tokens are chosen by their semantic similarity to the categories rather than randomly. further, it employs a new pretraining task: the prediction of which categories appear in the input text. notice that unlabeled text does not contain supervision for our pre-training tasks and we hence have to use distant supervision as an approximation. in our unsupervised da experiments we consider laptop and restaurant reviews, where for the restaurant domain we consider two variants, that differ in their difficulty. our best performing model outperforms the strongest baseline by over 5% on average and over 13% on the most challenging setup, while using only a small fraction of the unlabeled data. moreover, we show that our pretraining procedure is very effective in resourcepoor setups, where unlabeled data is scarce. ## background and previous work while both aspect extraction and domain adaptation are active fields of research, research on their intersection is not as frequent. we hence first describe in-domain ae, and then continue with a survey of da, focusing on pivot-based unsupervised da. finally, we describe works at the intersection of both problems. ## domain adaptation with dilbert in this section, we introduce dilbert, a new finetuning scheme for bert @xcite , customized for domain and category shift. recall that bert performs two pre-training tasks: (a) masked language modeling (mlm), where some of the input tokens are randomly masked and the model should predict them based on their context; and (b) next sentence prediction (nsp), where the model is provided with sentence pairs from its training data and it should predict whether one sentence is indeed followed by the other. dilbert modifies the mlm task and presents a new pre-training task. notice that dilbert is applied to a bert model that has already been trained on general text -text that is not directly related to the source and target domains of interest -with the standard mlm and nsp tasks. hence, dilbert can be seen as a fine-tuning step on the unlabeled data from the source and target domains. after dilbert is applied, a task classifier is added on top of the resulting bert model and this model is fine-tuned on the labeled source domain data to perform that aspect extraction task. this final model will eventually be applied to test data from the target domain. we next describe the two pretraining tasks of dilbert. category-based masked language modeling (cmlm) as noted above, ben-david et al. (2020) integrated pivot-based training into the mlm pre-training task of bert, in order to facilitate da for sentiment classification. as noted in §1, it is challenging to define pivot features for the ae task, both because (a) it is a sequence-tagging, token-level task, and because (b) the aspect words and their categories tend to change across domains. both these properties challenge the "high mi with the task label" criterion in the pivot definition (criterion (b) of § 1), as this criterion is designed for (1) sentence (or larger text) labels where words can correlate with the task label (and hence property (a) above is challenging); and where (2) the correlation with the task label in the source domain is indicative of the correlation in the target domain (and hence property (b) is challenging). to alleviate these limitations, we present the category-based masked language model (cmlm) task: a variant of the bert mlm pre-training task (left part of figure 2 ). cmlm operates similarly to mlm, but with one difference: while in standard mlm the masked tokens are randomly chosen, cmlm harnesses information about the aspect categories in the source and target domains in order to mask words that are more likely to bridge the gap between the domains. 2 we start by training static (non-contextualized) word embeddings on unlabeled data from both the source and the target domains. then, we iterate over the input text and compute the cosine similarity between each input word and the word em-figure 2 : illustrations of dilbert fine-tuning tasks, cmlm and cpp, for an example input sentence. first, we calculate a similarity score for each (word, category) pair using their pre-trained, static word embeddings. then, for each word and for each category, we keep only the maximal score. for cmlm (left), we mask the top α% words, according to their score. for cpp (right), we construct a label vector in which the i th coordinate is set to 1 if the i th category score is greater than β, a threshold hyper-parameter, and to 0 otherwise. fc is a fully connected layer. the tasks are trained sequentially: first cmlm and then cpp. ## experimental setup task and domains we experiment with the task of cross-domain ae @xcite . we consider data from the amazon laptop reviews (l) and the yelp restaurant reviews (r) domains. the labeled data from the l domain is taken from the semeval 2014 task on absa @xcite . we follow gong et al. (2020) and combine the semeval 2014, 2015 and 2016 restaurant datasets @xcite @xcite into a single dataset, removing all duplicated examples. this results in 3045/800 and 3845/2158 train/validation examples for the l and r domains, respectively. at test time the target domain validation set serves as a test set. we obtain the list of category names for each domain from @xcite . 5 we used 863,000 laptop reviews from the amazon reviews dataset of @xcite , and 570,000 reviews from the yelp open dataset foot_4 as our unlabeled dataset for both domains, respectively. to consider a more challenging setup, we experiment with the mams dataset @xcite , consisting of 5297 labeled reviews from the restaurant domain (m). we used 4297 reviews as a training set and 1000 as a validation set, and the same unlabeled data as for the r domain. each review in the mams dataset has at least two aspects with different sentiment polarities, making it harder to adapt to, as the label distribution is different from that of the semeval datasets. we consider adaptation from the l to the m domain and vice versa, adding two source-target pairs to our experiments (the m and r domains both address the same topic, and we consider them too similar for adaptation). ## conclusions and future work we have presented dilbert, a customized pretraining approach for unsupervised da with category shift, and apply it to the task of aspect extraction. we demonstrate that by fine-tuning with a modified version of the bert pre-training tasks, we can better adapt to new domains and aspect categories, even in resource-poor scenarios where unlabeled data is limited. to make our experimentation more challenging, we presented a new ae domain (m), which is substantially different from previously presented ones. in future work, we would like to extend our approach so that it can jointly solve the aspect extraction and sentiment analysis tasks (the absa task). moreover, we would like to verify the quality of our approach in additional domains, tasks (i.e., going beyond ae to other tasks that present category shifts when domains change) and eventually even languages. ying ding, jianfei yu, and jing jiang. 2017. recurrent neural networks with auxiliary labels for crossdomain opinion target extraction. in proceedings of the aaai conference on artificial intelligence, volume 31. timothy dozat and christopher d. manning. 2017. deep biaffine attention for neural dependency parsing. in iclr. yaroslav ganin, evgeniya ustinova, hana ajakan, pascal germain, hugo larochelle, françois laviolette, mario marchand, and victor lempitsky. 2016. domain-adversarial training of neural networks. the journal of machine learning research, 17(1):2096-2030. chenggong gong, jianfei yu, and rui xia. 2020. unified feature and instance based domain adaptation for aspect-based sentiment analysis. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 7035-7045, online. association for computational linguistics. niklas jakob and iryna gurevych. 2010. extracting opinion targets in a single and cross-domain setting with conditional random fields. in proceedings of the 2010 conference on empirical methods in natural language processing, pages 1035-1045.