Domain Adaptation for Sentiment Analysis Using Robust Internal Representations (2023)

Abstract: Sentiment analysis is a costly yet necessary task for enterprises to study the opinions of their customers to improve their products and to determine optimal marketing strategies. Due to the existence of a wide range of domains across different products and services, cross-domain sentiment analysis methods have received significant attention. These methods mitigate the domain gap between different applications by training cross-domain generalizable classifiers which relax the need for data annotation for each domain. We develop a domain adaptation method which induces large margins between data representations that belong to different classes in an embedding space. This embedding space is trained to be domain-agnostic by matching the data distributions across the domains. Large interclass margins in the source domain help to reduce the effect of “domain shift” in the target domain. Theoretical and empirical analysis are provided to demonstrate that the proposed method is effective.

## introduction the main goal in sentiment classification is to predict the polarity of users automatically after collecting their feedback using online shopping platforms, e.g., amazon customer reviews. a major challenge for automatic sentiment analysis is that polarity is expressed using completely dissimilar terms and phrases in different domains. for example, while terms such as "fascinating" and "boring" are used to describe books, terms such as "tasty" and "stale" are used to describe food products. as a result of this discrepancy, an nlp model that is trained for a particular domain may not generalize well in other different domains, referred as the problem of "domain gap" @xcite . since generating annotated training data for all domains is expensive and time-consuming @xcite , cross-domain sentiment analysis has gained significant attention recently @xcite @xcite @xcite @xcite @xcite . the goal is to relax the need for annotation via transferring knowledge transfer from another domain with annotated data to train models for domains with unannotated data. the above problem has been studied more broadly in the domain adaptation (da) literature @xcite . a common da approach is to align the distributions of both domains in a shared embedding space @xcite . as a result, a source-trained classifier that receives its input from the embedding space will generalize well in the target domain. in the sentiment analysis problem, this means that polarity of natural language can be expressed independent of the domain in the embedding space to transcendent discrepancies. we can model the embedding space using a shared deep encoder trained to align the distributions at its output space. this training procedure has been implemented indirectly using adversarial learning @xcite @xcite or by directly minimizing loss functions that are designed to align two probability distributions @xcite @xcite @xcite . contributions: our idea is based on learning a parametric distribution for the source domain in a cross-domain embedding space @xcite . we estimate this distribution as a gaussian mixture model (gmm). we then use the gmm distribution to align the source and the target distributions using confident samples, drawn for them gmm to increase the interclass margins to reduce the domain gap. ## related work while domain adaptation methods for visual domains usually use generative adversarial networks (gans) @xcite ,the dominant approach for cross-domain sentiment analysis is to design appropriate loss functions that directly impose domain alignment. the main reason is that natural language is expressed in terms of discrete values such as words, phrases, and sentences. since this domain is not continuous, even if we convert natural language into real-valued vectors, it will not be differentiable. hence, adversarial learning procedure cannot be easily adopted for natural language processing (nlp) applications. the alternative approach is to minimize a probability distribution metric to reduce domain cap @xcite . for example, using wasserstein distance (wd) for domain alignment in visual domains has been found to be highly effective @xcite @xcite . we rely on the sliced wasserstein distance (swd) for aligning distribution because of having a less computational load @xcite . the major reason for performance degradation of a source-trained model in a target domain stems from "domain shift", i.e., the boundaries between the classes change in the embedding space which in turn increases possibility of misclassification. it has been shown that if an increased-margin classifier is trained in the source domain, it can generalize better than many methods that try to only align distributions @xcite . inspired by this argument, our method is based on both aligning distributions in the embedding space and also inducing larger margins between classes by learning a "parametric distribution" for the source domain. our idea is based on the empirical observation that when a deep network classifier is trained in a domain with annotated data, data points of classes form separable clusters in the embedding space modeled via the network responses in hidden layers. this means that the source distribution can be modeled as a multimodal distribution in the embedding space. our work is based on using the multimodal distribution to induce larger margins between the classspecific clusters after an initial training phase in the source domain. ## cross-domain sentiment analysis consider two sentiment analysis problems in a source domain s with an annotated dataset d @xmath0, y s ), where given a family of parametric functions f θ : r @xmath1, e.g., deep neural networks with learnable weights θ, and considering an ideal labeling function f (•), e.g., ∀(x, y) : @xmath2, the goal is to search for the optimal predictor f θ * (•) in this family for the target domain. this model should have minimal expected error, i.e., given a large enough labeled dataset in the source domain, erm model generalizes well in the source domain. the source-trained model may also perform better than chance in a similar target domain, given cross-domain knowledge transfer, yet its performance will degrade in the target domain compared to its performance in the source domain because of existing distributional discrepancy between the two domains, p s ̸ = p t . our goal is to benefit from the encoded information in the unlabeled target domain data points and adapt the source-trained classifier f θ to generalize better in the target domain. we use the common approach of reducing the domain gap across the two domains by mapping data points into a shared embedding. we consider that the predictor model f θ (•) can be decomposed into a deep encoder subnetwork ϕ v (•) : @xmath3: @xmath4, where @xcite , the internally learned multimodal distribution is estimated using a gmm. right: random samples from the gmm with high-confident labels are drawn and then used to generate a pseudo-dataset. the pseudo-dataset helps to induce larger margins between the classes to mitigate domain shift in the target domain. ## increasing interclass margins our idea for increasing margins between the classes is based on learning an intermediate parametric distribution in the embedding space. we demonstrate that this distribution can be used to induce larger margins between the classes. to this end, we consider that the classifier subnetwork consists of a softmax layer. this means that the classifier should become a maximum a posteriori (map) estimator after training to be able to assign a membership probability to a given input feature vec- training on the source domain: use (4) and estimate α j , µ j , σ j 8: model adaptation: update the model by solving (6) 19: end for tor. under this formulation, the model will generalize in the source domain if after supervised training of the model using the source data, the input distribution is transformed into a multi-modal distribution p j (•) = ϕ v (p s )(•) with k modes in the embedding space (see figure 1 , left). each mode of this distribution corresponds to one type of sentiments. the geometric distance between the modes of this distribution corresponds to the margins between classes. if we test the sourcetrained model in the target domain, the boundaries between class modes will change due to "domain shift", i.e., ϕ v (p t )(•) ̸ = ϕ v (p s )(•). as visualized in figure 1 , if we increase the margins between the class-specific modes in the source domain, domain shift will likely cause less performance degradation @xcite . we estimate the multimodal distribution in the embedding as a gaussian mixture model (gmm): to cancel out outliers, we include only those source samples in the s j sets, for which the source-trained model predicts the corresponding labels correctly. the closed-form map estimate for the mode parameters is given as: computations in eq. ( 4 ) can be done efficiently. for a complexity analysis, please refer to the appendix (section c). our idea is to use this multimodal gmm distributional estimate to induce larger margins in the source domain (see figure 1 , right). we update the domain alignment term in (2) to induce larger margins. to this end, we augment the source domain samples in the domain alignment term with samples of a labeled pseudodataset d @xmath5, y p ) that we generate using the gmm estimate, where z @xmath6, . . . , z p np ] ∈ r p×np , y @xmath7, ..., y p np ] ∈ r k×np . this pseudo-dataset is generated using the the gmm distribution. we draw samples from the gmm distributional estimate z p i ∼ pj (z) for this purpose. to induce larger margins between classes, we feed the initial drawn samples into the classifier network and check the confidence level of the classifier about its predictions for these randomly drawn samples. we set a confidence threshold level τ ≈ 1 and only select a subset of samples for which the confidence level of the classifier is more than τ : given the gmm distributional form, selection of samples based on the threshold τ means that we include gmm samples that are closer to the classspecific mode means µ i (see figure 1 ). the margins between the clusters in the source domain increase if we use the generated pseudo-dataset for domain alignment. hence, we update eq. ( 2 ) as: the first and the second terms in (6) are erm terms for the source dataset and the generated pseudodataset to guarantee that the classifier continues to generalize well in the source domain after adaptation. the third and the fourth terms are empirical swd losses (see appendix for more details) that align the source and the target domain distributions using the pseudo-dataset which as we describe induces larger margins. the hope is that as visualized in figure 1 , these terms can reduce the effect of domain shift. our proposed solution, named sentiment analysis using increased-margin models (saim 2 ), is presented in algorithm 1 and figure 1 . ## theoretical analysis following a standard pac-learning framework @xcite , we prove that algorithm 1 minimizes an upperbound for the target domain expected error. similarly, we can build an empirical distribution for the multimodal distribution μp = 1 np np @xmath8. in our analysis we also use the notion of jointoptimal model h s,t (•) which is defined as: w * = arg min w e s,@xmath9. when we have labeled data in both domains, this is the best erm-trained model. existence of a good joint-trained model guarantees that the domains are related for positive transfer, e.g., similar sentiment polarities are encoded across the two domains. theorem 1: consider that we use the procedure described in algorithm 1 for cross-domain sentiment analysis, then the following inequality holds for the target expected error: proof: the complete proof is included in the appendix (section c). theorem 1 provides an explanation to justify algorithm 1. we observe that all the terms in the upperbound of the target expected error in the righthand side of (7) are minimized by saim 2 . the source expected error is minimized as the first term in (6). the second and the third terms are minimized as the third and fourth terms of (6). the fourth term 1τ will be small if we set τ ≈ 1. the term e s,p is minimized through the first and the second term of (6). this is highly important as using the pseudo-dataset provides a way to minimize this term. as can be seen in our proof in the appendix, if we don't use the pseudo-dataset, this terms is replaced with e s,t which cannot be minimized directly due to lack of having annotated data in the target domain. the last term in ( 7 ) is a constant term that as common in pac-learning can become negligible states that in order to train a good model if we have access to large enough datasets. hence all the terms in the upperbound are minimized and if this upperbound is tight, then the process leads to training a good model for the target domain. if the two domain are related, e.g., share the same classes, and also classes become separable in the embedding space, i.e., gmm estimation error for the source domain distribution in the embedding space is small, then the upperbound is going to be likely tight. however, we highlight that the prospect of a tight upperbound is a condition for our algorithm to work in practical settings. note, however, this is a common limitation for most parametric machine learning algorithms. ## experimental validation we have selected the most common setup for uda sentiment analysis to perform our experiments for possibility of comparing our performance against prior works. our implemented code is publicly available: https://github.com/digbose92/saim2 6.1 experimental setup dataset and the cross-domain tasks: most existing works in cross-domain sentiment analysis re-port performance on cross-domain tasks that are defined using the real-world amazon reviews benchmark dataset @xcite . the dataset is built using amazon product reviews from four product domains: books (b), dvd (d), electronics (e), and kitchen (k) appliances. hence, 12 pairwise cross-domain tasks can be defined amongst the domains. each review is considered to have positive (higher than 3 stars) or negative (3 stars or lower) sentiment. each task consists of 2000 labeled reviews for the source domain and then 2000 unlabeled reviews for the target domain, and 2500-5500 examples for testing. we report our performance on the 12 definable cross-domain tasks. we report the average prediction accuracy and standard deviation (std) over 5 runs of our code. preprocessing: we have used two methods for backbone feature extraction: tf-idf and bert features @xcite . the most common setup in the literature is using tf-idf feature vector of bag-of-words unigrams and bigrams, where we encode each review as a @xmath10. following @xcite , qe also report performance when modern bert features are used. following the precedence, we first fine-tune a bert backbone, followed by fcc layers, on the source domain to solve the sentiment analysis task. we call this baseline vanilla bert (vbert), analogues to so for tf-idf features. features extracted via vbert are then used by the bert-based methods for domain adaptation. note that using bert as an advanced text embedding method increases the absolute performance, i.e., pretraining of bert on a large auxiliary dataset enables knowledge transfer. however, in model adaptation we must evaluate the algorithms according to the relative improvement of the model performance on the target domain compared to the solely source-trained model. hence, only relative improvement over the baseline should be used for evaluating performance of domain adaptation algorithms. ## conclusions we developed a method for cross-domain sentiment analysis based on aligning two domain-specific dis-tributions in a shared embedding space and inducing larger margins between the classes using an intermediate multi-modal gmm distribution. we theoretically demonstrated that our approach minimizes an upperbound for the target domain error. our experiments demonstrate that our algorithm is effective and compares favorably against the stateof-the-art. a future research direction is to address cross-domain sentiment analysis when distributed source domains exist @xcite .