Unsupervised Reinforcement Adaptation for Class-Imbalanced Text Classification (2022)

Abstract: Class imbalance naturally exists when label distributions are not aligned across source and target domains. However, existing state-of-the-art UDA models learn domain-invariant representations across domains and evaluate primarily on class-balanced data. In this work, we propose an unsupervised domain adaptation approach via reinforcement learning that jointly leverages feature variants and imbalanced labels across domains. We experiment with the text classification task for its easily accessible datasets and compare the proposed method with five baselines. Experiments on three datasets prove that our proposed method can effectively learn robust domain-invariant representations and successfully adapt text classifiers on imbalanced classes over domains.

## introduction unsupervised domain adaptation (uda) is to find a shared feature space that is predictive across target and source domains @xcite . the shared space, domain-independent feature set, allows transferring of trained text classifiers from the source domain to the target domain. methods to find the space have two major directions, pivot feature @xcite @xcite and adversarial learning @xcite @xcite . the pivotbased method selects a subset of shared features, called pivots, which learn important cross-domain information to represent shared feature space. adversarial learning approaches the shared feature space by reducing document features' capability to distinguish source and target domains. the common method to achieve this is gradient reversal layer (grl) @xcite aiming to reduce domain-specific patterns. however, the uda approaches primarily focus on feature shifts (𝑝(𝑋 𝑠𝑜𝑢𝑟 𝑐𝑒 )! = 𝑃(𝑋 𝑡 𝑎𝑟 𝑔𝑒𝑡 )) while ignore possible class shifts (𝑝(𝑌 𝑠𝑜𝑢𝑟 𝑐𝑒 )! = 𝑝(𝑌 𝑡 𝑎𝑟 𝑔𝑒𝑡 )) across domains. class-imbalance naturally exists in data when label distributions across domains @xcite are different. under the classimbalanced scenario, the label distribution is imbalanced across domains, and the label distributions in source and target domains are not the same. given the widely used amazon data @xcite as an example, the book reviews may have more positive reviews than negative reviews, and the kitchen may have a lower ratio of negative reviews. however, evaluating unsupervised domain adaptation under the class-imbalanced scenario is under-examined than the ideal scenario of the class-balanced benchmark. a wide evaluation benchmark of uda for text classifiers is extracted from the amazon review @xcite . the data has the same balanced-class distributions for both source and target domains. such a well-balanced label distribution may make existing uda models inapplicable to the real-world scenario, where class distributions can shift across domains. in this study, we proposed an unsupervised reinforcement adaptation model (uram) for text classifiers under the uda setting that only labeled source data and unlabeled target data are available. specifically, we propose a neural mask mechanism to generate domain-dependent and -independent feature representations and a reward policy using a critic value network @xcite (crn) to learn optimal domain-independent representations. the reward policy optimizes the uram via three joint reward factors, label, domain, and domain distance. while the label reward aims to encourage text classification models on domainindependent features to predict correct document classes, the domain and domain distance rewards reduce domain variations of domain-dependent feature representations between source and target domains. we compare our reinforcement adaptation model with five baselines and experiment on four class-imbalanced data with both binary and non-binary labels. the results using the f1-score demonstrate the effectiveness of our reinforcement learning model that outperforms the baselines by 3.13 on average. the main contributions of this paper are as follows: this section briefly recaps the concepts of unsupervised domain adaptation (uda) and reinforcement learning. 2.1 uda for class-imbalanced data uda assumes a labeled dataset with d 𝑆 = 𝑥 𝑖 𝑠 , 𝑦 𝑖 𝑠 𝑛 𝑠 𝑖=1 from source domain and a unlabeled data d 𝑇 = 𝑥 𝑗 𝑡 𝑛 𝑡 𝑗=1 from target domain, data distributions of the two domains are different, 𝑝(𝑥 𝑠 ) ≠ 𝑝(𝑥 𝑡 ), and the two domains share the same number of unique annotations. uda is to find a common feature space aligning source and target domains so that 𝑓 ( 𝑝(𝑥 𝑠 )) ≈ 𝑝(𝑥 𝑡 ) however, class-imbalanced data naturally exist in uda tasks that may cause inefficient knowledge transfer @xcite . we assume both data and labels are not equally distributed in this work. ## related work unsupervised domain adaptation for text classification has several major types of approaches, pivot features @xcite @xcite , instance weighting @xcite @xcite , and domain adversaries @xcite @xcite . a recent survey @xcite shows that the most widespread methods for neural uda are based on the use of domain adversaries, which reduces the discrepancy between the source and target distributions by reversing gradient updates for domain prediction networks. our study follows the same track to obtain domain-invariant representations, however, there reinforcement learning with the robustness in learning sophisticated policies, recent works introduce reinforcement learning (rl) into the unsupervised domain adaptation task @xcite @xcite . darl @xcite employs deep qlearning in partial domain adaptation. the darl framework designs a reward for the agent-based on how relevant the selected source instances are to the target domain. with the action-value function optimizer, darl can automatically select source instances in the shared classes for circumventing negative transfer as well as to simultaneously learn transferable features between domains by reducing the domain shift. however, darl does not generalize to unsupervised domain adaptation. highly relying on the rich labels in the source domain will cause failure when insufficient labels are in the target domain. to address this problem, zhang et al. develop a new reward across both source and target domains. this reward can guide the agent to learn the best policy and select the closest feature pair for both domains. however, rarely study has deployed the reinforcement uda into the class-imbalanced text classification. to our best knowledge, we are the first work introducing rl for the uda under the class-imbalanced text classification. imbalanced-class increasing works study the class-imbalanced domain adaptation @xcite @xcite . coal @xcite deals with feature shift and label shift in a unified way. with the idea of prototype-based conditional distribution alignment and class-balanced self-training, coal tackles feature shift in the context of label shift. however, present works only focus on computer vision, and the imbalanced class domain adaptation in nlp is unexplored. the other similar works is category-level feature alignment @xcite @xcite @xcite . these works usually focus on domain shifts and propose domain-level aligned strategies while ignoring the local category-level distributions, reducing cross-domain text classifiers' effectiveness. a popular strategy for category-level alignment is aligning the same class features among different domains respectively by resorting to pseudo labels @xcite . ## conclusion in this study, we have proposed an unsupervised reinforcement adaptation model (uram) for the novel cross-domain adaptation challenge where the source and target domains are class-imbalanced. we demonstrate the effectiveness of our reinforcement approach with the other four state-of-art baselines on the task of text classification. the uram learns domain-independent representations by leveraging three reward factors, label, domain, and domain distance, which coherently combines pivot and adversarial approaches in uda. extensive experiments and ablation analysis show that the uram can obtain robust domain-invariant representations and effectively adapt text classifiers over both domains and imbalanced data.