Conditional Adversarial Networks for Multi-Domain Text Classification (2021)

Abstract: In this paper, we propose conditional adversarial networks (CANs), a framework that explores the relationship between the shared features and the label predictions to impose stronger discriminability to the learned features, for multi-domain text classification (MDTC). The proposed CAN introduces a conditional domain discriminator to model the domain variance in both the shared feature representations and the class-aware information simultaneously, and adopts entropy conditioning to guarantee the transferability of the shared features. We provide theoretical analysis for the CAN framework, showing that CAN’s objective is equivalent to minimizing the total divergence among multiple joint distributions of shared features and label predictions. Therefore, CAN is a theoretically sound adversarial network that discriminates over multiple distributions. Evaluation results on two MDTC benchmarks show that CAN outperforms prior methods. Further experiments demonstrate that CAN has a good ability to generalize learned knowledge to unseen domains.

## introduction text classification is a fundamental task in natural language processing (nlp) and has received constant attention due to its wide applications, ranging from spam detection to social media analytics @xcite @xcite @xcite . over the past couple of decades, supervised machine learning methods have shown dominant performance for text classification, such as naive bayes classifiers @xcite , support vector machines @xcite and neural networks @xcite . in particular, with the advent of deep learning, neural network-based text classification models have gained impressive achievements. however, text classification is well known to be highly domain-dependent, the same word could convey different sentiment polarities in different domains @xcite . for example, the word infantile expresses neutral sentiment in baby product review (e.g., the infantile cart is easy to use), while in book review, it often indicates a negative polarity (e.g., this book is infantile and boring). thus a text classifier trained on one domain is likely to make spurious predictions on another domain whose distribution is different from the training data distribution. in addition, it is always difficult to collect sufficient labeled data for all interested domains. therefore, it is of great significance to explore how to leverage available resources from related domains to improve the classification accuracy on the target domain. the major line of approaches to tackle the above problem is multi-domain text classification (mdtc) @xcite , which can handle the scenario where labeled data exist for multiple domains, but in insufficient amounts to training an effective classifier. deep learning models have yielded impressive performance in mdtc @xcite . most recent mdtc methods adopt the shared-private paradigm, which divides the latent space into two types: one is the shared feature space for all domains with the aim of capturing domain-invariant knowledge, the other one is the private feature space for each domain which extracts domain-specific knowledge. to explicitly ensure the optimum separations among the shared latent space and multiple domain-specific feature spaces, the adversarial training @xcite is introduced in mdtc. by employing the adversarial training, the domain-specific features can be prevented from creeping into the shared latent space, which will lead to feature redundancy @xcite . in adversarial training, a multinomial domain discriminator is trained against a shared feature extractor to minimize the divergences across different domains. when the domain discriminator and the shared feature extractor reach equilibrium, the learned shared features can be regarded as domain-invariant and used for the subsequent classification. the adversarial training-based mdtc approaches yield the stateof-the-art results @xcite . however, these methods still have a significant limitation: when the data distributions present complex structures, adversarial training may fail to perform global alignment among domains. such a risk comes from the challenge that in adversarial training, only aligning the marginal distributions can not sufficiently guarantee the discriminability of the learned features. the features with different labels may be aligned, as shown in figure 1 . the critical mismatch can lead to weak discriminability of the learned features. in this paper, motivated by the conditional generative adversarial networks (cgans), which aligns distributions of real and generated images via conditioning the generator and discriminator on extra information @xcite , we propose conditional adversarial networks (cans) to address the aforementioned challenge. the can method introduces a conditional domain discriminator that models domain variance in both shared features and label predictions, exploring the relationship between shared feature representations and class-aware information conveyed by label predictions to encourage the shared feature extractor to capture more discriminative information. moreover, we use entropy conditioning to avoid the risk of conditioning on the class-aware informa-tion with low certainty. the entropy conditioning strategy can give higher priority to easy-to-transfer instances. we also provide a theoretical analysis demonstrating the validity of cans. our approach adopts the shared-private paradigm. we validate the effectiveness of can on two mdtc benchmarks. it can be noted that can outperforms the state-of-the-art methods for both datasets. finally, we empirically illustrate that can has the ability to generalize in cases where no labeled data exist for a subset of domains. the contributions of our work are listed as follows: ## related work multi-domain text classification (mdtc) was first proposed by @xcite , aiming to simultaneously leverage all existing resources across different domains to improve system performance. currently, there are two main streams for mdtc: one strand exploits covariance matrix to model the relationship across domains @xcite @xcite ; the other strand is based on neural networks, sharing the first several layers for each domain to extract low-level features and generating outputs with domain-specific parameters. the multi-task convolutional neural network (mt-cnn) utilizes a convolutional layer in which only the lookup table is shared for better word embeddings @xcite . the collaborative multi-domain sentiment classification (cmsc) combines a classifier that learns common knowledge among domains with a set of classifiers, one per domain, each of which captures domain-dependent features to make the final predictions @xcite . the multi-task deep neural network (mt-dnn) maps arbitrary text queries and documents into semantic vector representations in a low dimensional latent space and combines tasks as disparate as operations necessary for classification @xcite . pioneered by the generative adversarial network (gan) @xcite , adversarial learning has been firstly proposed for image generation. @xcite applies adversarial learning in domain adaptation to extract domain-invariant features across two different distributions (binary adversarial learning). @xcite extends it to multiple adversarial learning, enabling the model to learn domain-invariant representations across multiple domains. however, only considering domaininvariant features can not provide optimal solutions for mdtc, because domain-specific information also plays an important role in training an effective classifier. @xcite proposes the shared-private paradigm to combine domaininvariant features with domain-specific ones to perform classification, illustrating that this scheme can improve system performance. to date, many state-of-the-art mdtc models adopt the adversarial learning and the shared-private paradigm. the adversarial multi-task learning for text classification (asp-mtl) utilizes long short-term memory (lstm) without attention as feature extractors and introduces orthogonality constraints to encourage the shared and private feature extractors to encode different aspects of the inputs @xcite . the multinomial adversarial network (man) exploits two forms of loss functions to train the domain discriminator: the least square loss (man-l2) and negative log-likelihood loss (man-nll) @xcite . the multi-task learning with bidirectional language models for text classification (mt-bl) introduces language modeling as an auxiliary task to encourage the domainspecific feature extractors to capture more syntactic and semantic information, and a uniform label distribution-based loss constraint to the shared feature extractor to enhance the ability to learn domain-invariant features @xcite . adversarial learning has several advantages, such as markov chains are not needed and no inference is required during learning @xcite . however, there still exists an issue in adversarial learning. when data distributions embody complex structures, adversarial learning can fail in performing the global alignment. the conditional generative adversarial network (cgan) is proposed to address this problem @xcite . in cgan, both the generator and discriminator are conditioned on some extra information, such as labels or data from other modalities, to yield better results. the conditional adversarial training mechanism has been explored in transfer learning. the conditional domain adversarial networks (cdans) condition the domain discriminator on a multilinear map of feature representations and category predictions so as to enable discriminative alignment of multi-mode structures @xcite . the conditional generative adversarial networks for structured domain adaptation learn a conditional generator to transform the feature maps of source domain images as if they were extracted from the target domain, and a discriminator to encourage realistic transformations for the semantic segmentation of urban scenes @xcite . sharing some spirit of cgan, this paper extends conditional adversarial learning in mdtc, enabling a domain discriminator on the shared features while conditioning it on the class-aware information conveyed by the label predictions. moreover, in order to guarantee the generalizability of the learned features, we also utilize the entropy conditioning strategy. ## approach in this paper, we consider mdtc tasks in the following setting. assume there exist m domains {d i } m @xmath0. for each domain, both labeled and unlabeled samples are taken into consideration. specifically, d i contains two parts: a limited amount of labeled samples l @xmath1, y j )} l i @xmath2; and a large amount of unlabeled samples u @xmath3. the challenge of mdtc lies in how to improve the system performance of mapping the input x to its corresponding label y by leveraging all available resources across different domains. the performance is measured as the average classification accuracy across the m domains. ## experiments we evaluate the effectiveness of the can model on both mdtc and unsupervised multi-source domain adaptation tasks. the former refers to the setting where the test data falls into one of the m domains, and the latter refers to the setting where the test data comes from an unseen domain without labels. moreover, an ablation study is provided for further analysis of the can model. ## conclusion in this paper, we propose conditional adversarial networks (cans) for mdtc. this approach can perform alignment on joint distributions of shared lemma 1. for any given f s , {f i d } m @xmath4, the optimum conditional domain discriminator d * is: proof. for any given f s , {f i d } m @xmath5, the optimum here, we utilize the lagrangian multiplier for d * under the condition (12). we have: let ∇@xmath6, we have: from the two above equations, we have: in our model, a mini-max game is implemented to achieve the optimum: therefore, by the non-negativity and convexity of the kl-divergence, we can have the corollary: corollary 1. when d is trained to its optimum d * , j d * is m log m . the optimum can be obtained if and only if p 1 (f, c) = p 2 (f, c) = ... = p m (f, c) = p (f, c).