Memory-Based Invariance Learning for Out-of-Domain Text Classification (2023)

Abstract: We investigate the task of out-of-domain (OOD) text classification with the aim of extending a classification model, trained on multiple source domains, to an unseen target domain. Recent studies have shown that learning invariant representations can enhance the performance of OOD generalization. However, the inherent disparity in data distribution across different domains poses challenges for achieving effective invariance learning. This study addresses this issue by employing memory augmentations. Specifically, we augment the original feature space using key-value memory and employ a meta-learning-based approach to enhance the quality of the invariant representations. Experimental results on sentiment analysis and natural language inference tasks show the effectiveness of memory-based method for invariance learning, leading to state-of-the-art performance on six datasets.

## introduction text classification has made remarkable progress in recent years, thanks to the advancements in deep neural networks such as transformer @xcite and pretrained language models (plms) @xcite @xcite . however, these learning systems heavily rely on the assumption that the training and test sets come from the same domain. when there is a significant discrepancy between the test domain (also known as the target domain) and the training domains (also known as source domains), the performance of traditional learning systems suffers significant declines @xcite . domain generalization (dg) aims to address this out-of-domain (ood) problem, which is a practical and challenging issue, particularly when the labeled and unlabeled information of the target domains is unknown during the training phase. ## method as illustrated in figure 2 , the proposed model comprises (a) a vanilla transformer enhanced by (b) a key-value memory. furthermore, (c) the output layer is responsible for text classification, while (d) the domain discriminators handle domain classification tasks. the memory serves to enhance the feature representation and mitigate domain-specific feature distributions. to accomplish this, we employ a key-value memory bank that learns the appropriate feature augmentations (section 3.1). to address domain shifts through memory augmentations, we introduce an episodic training strategy (section 3.2). the training objective of the key-value memory can be formulated as bi-level optimization (section 3.3). lastly, we present the overarching meta-training and meta-test algorithms (section 3.4). ## experiments we evaluate the proposed method on sentiment analysis and natural language inference (nli) tasks. ## conclusion we have conducted an investigation into a memorybased approach for domain generalization (dg). our study involves the integration of a key-value memory network into the transformer model, and the proposal of a meta-learning algorithm that incorporates an episodic training strategy to effectively learn transferable memory for addressing domain shifts. the results obtained from experiments conducted on sentiment analysis and natural language inference tasks demonstrate the significant enhancement in transferability of the sourcedomain model through the usage of the memory unit. additionally, our approach achieves state-ofthe-art performance on six different datasets.