Domain Generalization for Text Classification with Memory-Based Supervised Contrastive Learning (2022)

Abstract: While there is much research on cross-domain text classification, most existing approaches focus on one-to-one or many-to-one domain adaptation. In this paper, we tackle the more challenging task of domain generalization, in which domain-invariant representations are learned from multiple source domains, without access to any data from the target domains, and classification decisions are then made on test documents in unseen target domains. We propose a novel framework based on supervised contrastive learning with a memory-saving queue. In this way, we explicitly encourage examples of the same class to be closer and examples of different classes to be further apart in the embedding space. We have conducted extensive experiments on two Amazon review sentiment datasets, and one rumour detection dataset. Experimental results show that our domain generalization method consistently outperforms state-of-the-art domain adaptation methods.

## introduction text classification is a highly important and widely studied natural language understanding task. recent success in self-supervised pre-training has significantly improved the state-of-the-art performance in sentiment analysis. however, sentiment classification is widely known as a domaindependent task, mainly because sentiment expressions can have different meanings in different domains (e.g., "long" in "long waiting time" of a restaurant review is negative, while in "long battery life" of a laptop review is positive). moreover, the amount of labeled data is highly imbalanced across different domains. many nlp domains still lack sufficient labeled data for training a high-performance classifier. therefore, it is crucial to adapt sentiment knowledge from resourcerich domains to low-resource domains. this strand of research is known as domain adaptation (da). prior works on domain adaptation typically follow a one-to-one @xcite or many-to-one adaptation setting @xcite , where the model is usually trained on labeled data from the source domain along with unlabeled data from the target domain and is then evaluated on the target domain data. the usage of the unlabeled target data is crucial in da. prior works mainly use it for domaininvariant representation learning or model selection. more recently, @xcite have demonstrated that large pretrained language models (prlms) are able to achieve promising performance for cross-domain sentiment classification. subsequent works further improve the adaptation performance through domain adversarial training @xcite , iterative pseudolabeling @xcite , or prompting @xcite , where they all adapt the da setting and assume unlabeled target data is available during model training. however, in more realistic scenarios, one may be asked to build a text classification model that will be applied to unknown target domains, which implies unlabeled target domain data is unavailable during training or model selection. domain generalization @xcite @xcite has been proposed to address this problem by learning a universal representation using labeled data from multiple source domains, without access to target domain data. building such a generalized model enables us to predict sentiment polarity of emerging domains, such as covid-19 vaccines and pandemic-related medical equipment. compared to domain adaptation, the major advantage of domain generalization is that one trained model can be used for all target domains, whereas domain adaptation needs to train a tailored model for each target domain. to the best of our knowledge, even though there are many works that focused on multi-source domain adaptation @xcite @xcite , there is no prior work that tackles domain generalization in the context of text classification. in this work, we focus on domain generalization for text classification. we aim to build a generalized sentiment classifier using labeled source data from multiple domains. the trained model can be applied to other unseen domains. we train a classifier to learn a joint hypothesis over all source domain data. on this basis, we propose to use supervised contrastive learning (scl) to better capture the similarity between examples from different domains but belong to the same sentiment class and contrast them with examples belonging to the other class. scl explicitly pulls the representations from the same class together and repulses the representations from different classes in the embedding space. this objective helps the model better extract the domain invariant features among all source domains, thereby allowing a better classification decision boundary. the usage of scl helps the classifier achieve better generalization ability, which is especially beneficial to the domain generalization scenario. although scl has previously been applied to image classification tasks @xcite and in-domain sentence-level classification tasks @xcite , to the best of our knowledge, we are the first to apply it in the context of domain generalization. it is shown that the performance of scl is highly affected by batch size as it requires a large number of contrasting examples for computing the contrastive loss @xcite ). an optimal batch size requires large memory, making it impractical in many use cases. to apply scl without excessive memory consumption, we further propose to use a memory bank to store the representations to increase the size of contrasting features, so that the hidden representations for sentiment classification will be reused for computing supervised contrastive loss. in this way, we can significantly improve performance compared to directly applying scl for text classification. to examine our proposed method, we conducted domain generalization experiments on two popular amazon review datasets (one monolingual and the other multilingual) and on the pheme rumour de-tection dataset. following previous works @xcite @xcite , different languages can be seen as distinct domains based on a shared cross-lingual encoder. hence, the domain generalization problem can also be extended to language generalization. we conducted experiments in cross-domain (cd), cross-language (cl), and cross-language crossdomain (clcd) settings. in our experiments, our proposed method is able to outperform the second best domain adaptation method by 0.81% accuracy score in sentiment analysis and 1.68% f1 score in rumour detection. our contributions can be summarized as follows: ## problem definition for the task of domain generalization on text classification, suppose we have labeled data from k source domains {d s i } k @xmath0. for each source domain, the labeled data d s i is denoted as d s @xmath1, y s i }. in the training phase, only source domains are available. then, the labeled dataset from the target domain d @xmath2, y t } is used for evaluation. the problem setup of domain generalization is different from multi-source domain adaptation (msda), which requires an additional unlabeled set from the target domain during training @xcite @xcite . in contrast, in the domain generalization setup, the model is only trained to obtain a domain-invariant feature from the given source domains, while the target-domain data d t is only used during evaluation. in this way, the trained model can be used to make predictions on unseen domains. ## model description since the domain generalization problem is to train an algorithm based on multiple source domains, the key challenge of this classification problem is to learn an ideal joint hypothesis of the source domains. that is, we aim to separate the data points by their labels as much as possible, thereby minimizing domain discrepancy within each class in the feature embedding space. our supervised contrastive learning model not only widens the margin of decision boundaries, but also enforces the distribution within the same class to be more uniform, hence minimizing the source domain distances for each class. we first shuffle all the source domain data and divide the joint dataset into mini-batches. this is mainly for enforcing a stable domain distribution for each mini-batch. for one sampled mini-batch of size n , we have @xmath3, y i } n @xmath4, where x i represents the input text, y i represents the sentiment label of that example. we first adopt a pretrained language model (prlm) as encoder. we use the hidden state of the last layer's [cls] token as document representation, and denote it as h. we then use a feed-forward neural network (followed by a tanh activation function and layer norm) f for dimension reduction. specifically, we have @xmath5. feature z will later be fed into a classifier g for downstream tasks: ## conclusions and future work in this paper, we study the under-explored domain generalization problem for text classification. we show that for cross-domain text classification, generalization performance from multiple source domains can exceed the best performance of one-toone domain adaptation, even if the target domain is unknown during training. to this end, domain generalization is more practical and easier to deploy in realistic scenarios. to further improve the performance of cross-domain text classification, we propose an effective and memory-saving approach based on supervised contrastive learning for the domain generalization problem. we conduct extensive experiments in cd, cl, and clcd settings. experimental results have shown that our framework consistently outperforms strong baselines and the previous state of the art in all three experimental settings.