Domain Generalization via Switch Knowledge Distillation for Robust Review Representation (2023)

Abstract: Applying neural models injected with in-domain user and product information to learn review representations of unseen or anonymous users incurs an obvious obstacle in content-based recommender systems. For the generalization of the in-domain classifier, most existing models train an extra plain-text model for the unseen domain. Without incorporating historical user and product information, such a schema makes unseen and anonymous users dissociate from the recommender system. To simultaneously learn the review representation of both existing and unseen users, this study proposed a switch knowledge distillation for domain generalization. A generalization-switch (GSwitch) model was initially applied to inject user and product information by flexibly encoding both domain-invariant and domain-specific features. By turning the status ON or OFF, the model introduced a switch knowledge distillation to learn a robust review representation that performed well for either existing or anonymous unseen users. The empirical experiments were conducted on IMDB, Yelp-2013, and Yelp-2014 by masking out users in test data as unseen and anonymous users. The comparative results indicate that the proposed method enhances the generalization capability of several existing baseline models. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/DG_RRR .

## introduction with the proliferation of social media, online shopping, and related activities, users are learning to provide an increasing number of reviews about the products they consume @xcite . with the deluge of customer reviews available, the sentiment score of *corresponding authors. to learn review representation, previous studies have sought to incorporate external user and product (up) information into sentiment analysis, which aims to build a neural model by learning contextual and external up features to predict rating scores @xcite @xcite . the main idea of these models is to inject up as external feature vectors into sentiment classifiers. these methods can be broadly categorized into two groups according to their injection strategies, i.e., bias-and matrix-based injections. bias-based methods render bias terms in classifier parameters, while matrix-based methods render matrix terms. bias-based methods typically perform weaker than matrix-based methods. however, such a method is hard to optimize and cumbersome @xcite . it regards these models as in-domain (id) distribution for specific users, which collects historical reviews of users on different products and uses them for recommendations. due to privacy preservation concerns, some users prefer to use anonymity to bypass the recommender system, which may understand users' intentions through historical interaction. unfortunately, applying neural models injected with id up information to learn review representations of unseen or anonymous users as out-of-domain (ood) distributions may degrade the classification performance for reviews. as shown in figure 1 , with the id up information, neural models could accurately classify positive and negative review samples. however, the performance drops when they face samples with anonymous up even though textual information is provided. we hypothesize that degradation occurs mainly because learning review representation depends heavily on external domain-specific features (f s ), ups from the recommender system. the proposed method masks out up information with a generalized up for sample augmentation. moreover, a kd strategy facilitate the generalization-domain representation to learn only domain-invariant features from review representation injected specific ups. finally, sentiment models can effectively handle with reviews from historical or unseen ups in inference. ## switch knowledge distillation in this section, we elaborate on the gswitch module and skd for dg problems in personalized sentiment analysis tasks, as shown in figure 2 . the gswitch model was proposed to adopt gswitch modules to inject id knowledge of up information into review representation and mask out id knowledge to simulate gd distribution. skd is a bidirectional kl divergence-based strategy to enhance gd representation with sufficient f i from id distribution. in practice, unseen or anonymous ood knowledge in the testing data can also be served with gd representation for dg tasks. moreover, such ood knowledge can be further updated for domain-specific knowledge if sufficient domain data are available. ## experiments to investigate the effectiveness of the proposed methods, extensive experiments were conducted on the review sentiment classification task. ## conclusions in this paper, a dg framework with knowledge distillation was proposed to generate robust review representations for sentiment analysis. rethinking the previous state-of-the-art models, we introduced gswitch models that connect review representations between id and gd distributions. to align both representations for sentiment classification, an skd was proposed, which enables id models to preserve f i for better generalization on ood data. comparative and analytical experiments indicate the effect of gswitch models and demonstrate that the proposed dg can effectively eliminate domain shifts in sentiment analysis.