Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis (2020)

Abstract: Cross-domain sentiment classification aims to address the lack of massive amounts of labeled data. It demands to predict sentiment polarity on a target domain utilizing a classifier learned from a source domain. In this paper, we investigate how to efficiently apply the pre-training language model BERT on the unsupervised domain adaptation. Due to the pre-training task and corpus, BERT is task-agnostic, which lacks domain awareness and can not distinguish the characteristic of source and target domain when transferring knowledge. To tackle these problems, we design a post-training procedure, which contains the target domain masked language model task and a novel domain-distinguish pre-training task. The post-training procedure will encourage BERT to be domain-aware and distill the domain-specific features in a self-supervised way. Based on this, we could then conduct the adversarial training to derive the enhanced domain-invariant features. Extensive experiments on Amazon dataset show that our model outperforms state-of-the-art methods by a large margin. The ablation study demonstrates that the remarkable improvement is not only from BERT but also from our method.

## introduction sentiment analysis aims to automatically identify the sentiment polarity of the textual data. it is an essential task in natural language processing with widespread applications, such as movie reviews and product recommendations. recently, deep networks have significantly improved the state-of-theart in sentiment analysis. however, training deep networks is highly depended on a large amount of labeled training data which is time-consuming and requires expensive manual labeling. thus, there is a strong need to leverage or reuse rich labeled data from a different but related source domain. cross-domain sentiment analysis, which transfers the knowledge learned from source domain to a new target domain, becomes a promising direction. the main challenge of cross-domain sentiment analysis is domain discrepancy due to different expression of the user's emotion across domains. to address the problem, a wide-used approach is designed to extract domain invariant features, which means that the distributions of features from the source domain and target domain are similar @xcite @xcite . one effective way to obtain domain-invariant features is adversarial training @xcite @xcite . specifically, a domain discriminator is learned by minimizing the classification error of distinguishing the source from the target domains, while a deep classification model learns transferable representations that are indistinguishable by the domain discriminator. very recently, pre-trained language models have shown to be effective for improving many language tasks @xcite . bidirectional encoder representations from transformers (bert) realized a breakthrough, which pre-trains its encoder using language modeling and by discriminating surrounding sentences in a document from random ones @xcite . pre-training in this manner can construct bidirectional contextual representation, and the large-scale pre-training enables bert powerful ability in language understanding. we only need to add one output layer and fine-tune bert to get the state-of-the-art results in sentiment analysis. theoretically, bert can enhance the performance in cross-domain sentiment analysis. however, some important problems remain when directly fine-tuning bert in the task of crossdomain sentiment analysis: firstly, there is no labeled data in the target do-main which brings many difficulties to the fine-tune procedure. if we fine-tune bert only by the source labeled data, the shift between training and test distributions will degrade the bert's performance. secondly, bert is task-agnostic and has almost no understanding of opinion text. bert is pretrained by the universal language wikipedia, leaving the domain challenges unresolved @xcite . for example, in the pre-training procedure, bert may learn to guess the @xcite in "the [mask] is bright" as "sun". but in a laptop sentiment analysis, it is more likely to be "screen". especially, in the cross-domain sentiment analysis scenario, the labeled data is limited, which is insufficient to fine-tune bert to ensure full domainawareness. thirdly, cross-domain sentiment analysis also arises a new challenge for bert to distinguish the characteristic of source and target domain to keep the transferable features and abandon domain-specific information. to address above problems, we design a novel pre-training task to make bert domain-aware and then improve the bert's fine-tuning procedure by adversarial training. specifically, a novel post-training procedure is implemented that adapts bert with unlabeled data from different domains to enhance the domain-awareness. apart from the target domain masked language model task, we introduce the domain-distinguish pre-training task. bert will be pre-trained to distinguish whether the two sentences come from the same domain. the domain-distinguish pre-training task will encourage bert to distill syntactic and semantic domain-specific features, so as to be domain-aware. the proposed post-training procedure gives us a new way to fully utilize language knowledge from the target domain and link the source and target in a self-supervised way. based on this, we could then conduct the adversarial training to derive the enhanced domain-invariant features. experiments on amazon reviews benchmark dataset show that our model gets the average result 90.12% in accuracy, 4.22% absolute improvement compared with state-of-the-art methods. the ablation study shows that the remarkable improvement is not only from bert but also from our method. the contributions of this paper can be summarized as follows: ## model in this section, we introduce the proposed model for cross-domain sentiment analysis in detail. we begin by giving the problem definition and notations. then bert and post-training method are formally presented in the second subsection. finally, the adversarial training process is introduced. we also give a theoretical analysis of our model. ## experiments in this section, we empirically evaluate the performance of our proposed methods. ## conclusion and future work in this paper, we propose the bert-daat model for cross-domain sentiment analysis. our purpose is to inject the target domain knowledge to bert and encourage bert to be domain-aware. specifically, we conduct post-training and adversarial training. a novel domain-distinguish pre-training task is designed to distill the domain-specific features in a self-supervised. experimental results on amazon dataset demonstrate the effectiveness of our model, which remarkably outperforms state-ofthe-art methods. the proposed post-training procedure could also be applied to other domain adaptation scenarios such as named entity recognition, question answering, and reading comprehension. in the future, we would like to investigate the application of our theory in these domain adaptation tasks.