Domain-Aware Contrastive Knowledge Transfer for Multi-domain Imbalanced Data (2022)

Abstract: In many real-world machine learning applications, samples belong to a set of domains e.g., for product reviews each review belongs to a product category. In this paper, we study multi-domain imbalanced learning (MIL), the scenario that there is imbalance not only in classes but also in domains. In the MIL setting, different domains exhibit different patterns and there is a varying degree of similarity and divergence among domains posing opportunities and challenges for transfer learning especially when faced with limited or insufficient training data. We propose a novel domain-aware contrastive knowledge transfer method called DCMI to (1) identify the shared domain knowledge to encourage positive transfer among similar domains (in particular from head domains to tail domains); (2) isolate the domain-specific knowledge to minimize the negative transfer from dissimilar domains. We evaluated the performance of DCMI on three different datasets showing significant improvements in different MIL scenarios.

## introduction the majority of existing works in imbalanced learning focus on the class imbalance setting where classes are presented in a long-tailed distribution: a subset of classes (head classes) have sufficient samples, while other uncommon or rare classes (tail classes) are underrepresented by limited samples. this setting is challenging because the model naturally focuses largely on the majority classes and there may be no sufficient data for tail classes to recover their underlying distribution @xcite . even though extensive work has been done on the class imbalance problem, the consideration of domains foot_0 is often missed. in many real-world scenarios, data naturally belongs to a set of domains e.g., for an online store, a potential domain assignment for each customer review can be defined based on the corresponding store departments. a simplistic solution is to ignore domain assignments and train a classifier for all domains, which we refer to as domain agnostic learning (d-al). d-al entirely ignores domains and assumes that the model can "automatically" discover the data distribution for domains and learn them equally well. the drawbacks of such an approach are obvious: if the training data is sourced from many domains, updating all parameters may lead the model to focus on the subsets of the data in proportion to their ease of access or frequency. moreover, if the data from different domains are dissimilar, agnostic learning may cause undesirable convergence dynamics i.e., negative transfer. we, therefore, argue that in the multi-domain imbalanced learning (mil) scenarios, a learning algorithm should consider domain information and leverage them to achieve effective knowledge transfer. the mil is a challenging problem. first, different domains may have very different number of samples and show a long-tailed distribution. for example, an intelligent assistant (e.g. amazon alexa) may provide a wide variety of skills and different skills may vary largely in number of examples. it is possible that some internal developed skills (e.g. music or whether) have hundreds of thousands of samples while many third-party developed skills may have only less than 10 samples in the same dataset @xcite . second, domains may exhibit different semantic similarities and disparities with each other. for instance, a feature may show positive correlation with a label for cer-tain domains while it is negatively correlated for others. third, the data-provided domain annotation may not be completely accurate or sufficiently finegrained. for example, a sentence "due to software or hardware issues, my computer cannot open my favorite text book, one hundred years of solitude" may belong to both computer and books domains while it may have only one domain assignment in the dataset. perhaps the most intuitive approach for mil is multi-task learning (mtl), where separate heads are used for different domains. while mtl considers domains, we will show it performs poorly in our experiments due to the lack of knowledge transfer between the classifiers. we believe that the key to successful mil is to not only enable but encourage positive transfer learning across domains. in this paper, we propose domain-aware contrastive knowledge transfer for muti-domain imbalance learning (dcmi). dcmi introduces a novel domain-aware representation layer based on domain embeddings which enables fine-grained and scalable representation sharing or separation. complementary to the data provided domain assignments, we use an auxiliary domain classification task to help determine the relevance of a sample to each domain i.e., soft domain assignments. dcmi uses a novel contrastive knowledge transfer objective to move the representation from similar domains closer and representation from dissimilar domains further apart. we conduct extensive experiments on three different multi-domain imbalanced datasets to demonstrate the effectiveness of dcmi. ## related work the recent imbalance learning literature can be organized into the following categories: data resampling. this is one of the most widely used practices to artificially balance the distribution. two popular options are under-sampling @xcite and over-sampling @xcite @xcite . under-sampling removes data from the head (dominant classes) while over-sampling repeats the data from the tail (minority classes). these approaches can be problematic as discarding tends to remove important samples and duplicating tends to introduce bias or overfitting. data augmentation. data augmentation has been used to enrich the tail classes. a popular approach is to leverage the mixup @xcite technique to augment the minority classes. remix @xcite assigns the label in favor of minority classes to the mixup samples, @xcite prepares a "feature cloud" for mixing up that has a larger distribution range for tail classes. @xcite adds noise to head classes to generate tail classes. @xcite decomposes the feature spaces and generate tail classes samples by combining class-shared feature from head classes and class-specific features from tail classes. however, this is usually a non-trivial work to generate meaningful samples that can help tail classes. loss reweighting. the basic idea of reweighting is to allocate larger weight for loss terms corresponding to tail classes while less weight for head classes. in class-sensitive cross-entropy loss @xcite , the weight for each class is inversely proportional to the number of samples. @xcite leverages a hold-out evaluation set to minimize the balanced loss. regularization. this approach adds an additional regularization term to improve the training for the tail samples. @xcite adds a factor to the standard cross-entropy loss to put more focus on hard, misclassified samples (usually attributed to the minority classes). @xcite proposed to regularize the minority classes strongly so that the generalization error of minority classes can be improved. while regularization is simple and effective, the soft penalty can be insufficient to make the model focus on the tail classes and a large penalty may negatively affect the learning itself. parameter isolation. it has been shown that decoupling the learning into representation learning and classifier learning can be quite effective. @xcite proposed a two-branch approach where the representation learning branch is trained as there is no class imbalance (input random sampling data) while the classifier learning branch applies the reverse sampling technique. the two branches are then combined by a curriculum learning strategy. @xcite further improves bbn by replacing the cross-entropy loss in representation learning branch into a prototypical supervised contrastive loss. this approach offers the opportunity to optimize each part separately but also make it hard to transfer knowledge from head to tail classes domain imbalanced learning. the above approaches mostly consider the class imbalance but ignore the imbalance across domains. cheng et al. ## problem definition in this paper, we assume access to a set of samples (x i , y i , j) for @xmath0. . . n }, y @xmath1. . . c}, and @xmath2. . . m }. here, n is the number of samples, c is the number of classes, and m is number of domains, i.e., shared feature space and label set across domains. we assume the scenario where exists (a) class imbalance: classes are not evenly distributed in each domain; (b) domain imbalance: domains are not evenly distributed, i.e., some domains may have much more or less number of examples than other domains; and (c) domain divergence: while some domains are naturally similar to others and thus positively correlated, some domains are naturally dissimilar to others and negatively correlated. given these assumptions, in multi-domain imbalanced learning (mil) we seek a model to minimize the expected loss for all domains (i.e., macro average). ## proposed method fig. 1 presents an overview of the proposed method. in the mil problem, it is crucial to identify the shared knowledge that can be transferred across similar domains to improve the tail domain performance and the domain-specific knowledge that needs to be handled carefully to avoid a negative transfer. to obtain domain-aware representations, we leverage domain embeddings to adaptively select the useful representation for each specific domain (sec. 4.1). additionally, regardless of the dataset provided domain assignment, in reality, a sample can belong to multiple domains to different degrees. to address this, we propose a domain classification task to obtain the relevance of a sample to each domain and transfer the related domain knowledge using a contrastive method (sec. 4.2). ## conclusion in this work, we studied the problem of learning from multi-domain imbalanced data, where not only there is class imbalance but also there is an imbalance among domains with varying degrees of similarity. we proposed a novel technique called dcmi that is capable of identifying the shared knowledge that can be transferred to improve the tail domain performance and the domain-specific knowledge that needs to be handled carefully to avoid negative transfer. dcmi employs a domainaware representation layer to adaptively select the relevant knowledge for each domain and leverages a novel contrastive learning objective to encourage knowledge transfer for relevant domains. based on the experiments using three challenging multi-domain imbalanced datasets, dcmi shows improvements over the current state-of-the-art and demonstrates applicability to different scenarios.