A Robust Information-Masking Approach for Domain Counterfactual Generation (2023)

Abstract: Domain shift is a big challenge in NLP. Many approaches, thus, resort to learning domain-invariant features to mitigate the hurdles of domain shift during inference. Such methods, however, inexorably fail to leverage the domain-specific nuances relevant to the task at hand. To avoid such drawbacks, domain counterfactual generation has recently been proposed that aims to transform a text from the source domain to a given target domain. To achieve this, the existing method uses a frequency-based approach to identify and mask the source-domain-specific tokens in a text. A pretrained LM is then prompted to fill the masks with target-domain-specific tokens. We, however, have observed that, due to limitations of the available data, such a frequency-based method may either miss some domain-token associations or lead to some spurious domain-token associations. To this end, we additionally employ attention norm-based scores to identify additional token-domain associations from a domain classifier. To minimize spurious associations, we also devise an iterative unmasking heuristic that unmasks the masked tokens to minimize the confidence of a domain classifier in the source domain. Our experiments empirically show that the counterfactual samples sourced from our masked text lead to improved domain transfer across various classification tasks. The proposed approach outperforms the baselines on 10 out of 12 domain-counterfactual classification settings with an average of 1.7% improvement in accuracy metric.

## introduction despite significant advances in unsupervised representation learning, natural language processing (nlp) systems often strongly rely on expensive human-annotated datasets. these (labeled) datasets, however, are only available in specific domains. systems trained on such datasets usually significantly under-perform on out-of-domain * equal contribution (ood) samples during inference, due to strong reliance on the dataset-specific token-or feature-label correlations. these correlations do not often generalize beyond the domain of the training dataset. these correlations can even be spurious annotation artifacts. for example, in a sentiment-labeled dataset on restaurant reviews, the token food may frequently appear in samples with negative sentiment score, due to selection bias @xcite . such biases are particularly prevalent in low resource settings, where training instances are scarce @xcite . in response, annotating samples from new domains may prove to be expensive and infeasible in the long term. to address these issues, many domain adaptations (da) techniques have been proposed @xcite @xcite @xcite . many of such techniques @xcite @xcite learn domaininvariant features that sidestep the pitfall of relying on domain-specific features when faced with ood inputs. this obviously also hinders performance on in-domain samples where domain-specific features may be relevant. to address this issue, @xcite recently proposed domaincounterfactual generation (docogen) to transform given in-domain samples into out-domain samples. docogen masks the source-domain-specific ngrams in the input using a token-frequency-based approach. these masks are filled with targetdomain-specific tokens using a conditional text generation language model that is fine-tuned with an unsupervised sentence reconstruction objective. however, the frequency-based masking approach does not account for the context to identify the source-domain-specific tokens. moreover, this approach is limited by the statistics of an incomplete training set. these shortcomings may cause the ap-proach to overlook many valid domain-token associations. for instance, in fig. 1 the word tearjerker is not masked by docogen, where this word may be associated with the dvd of a tragedy movie. furthermore, such frequency-based masking fails to generalize well to out-of-vocabulary words. therefore, we propose a robust masking method that can achieve better retention of non-domain information, while removing domain-specific information. our method includes three phases: frequency-based mask initialization @xcite , over-the-top (oot) masking, and unmasking. we first phase is simply to initialize the mask using the frequency-based strategy by @xcite . these masks are based on token-domain association consisting of two factors: the probability of the presence of a target token in a particular domain and the non-uniformity of this probability distribution. tokens surpassing a certain association score, w.r.t. both source and target domain, are masked. in the second phase, we leverage the encoded knowledge in language models (lm) @xcite to identify additional tokendomain associations, improving recall at the risk of hurting precision. we found that a token with high attention-norm @xcite , in a finetuned lm as domain classifier, could be strongly associated with the predicted domain. such tokens with high enough attention norms are masked. finally, we sequentially unmask the masked tokens, guided by the confidence score of a domain classifier. to minimize spurious token-domain associations from the prior two masking phases that hurt precision, the tokens that cause low domain confidence upon unmasking are kept unmasked. the overall contributions of this work are: i) a robust masking strategy to mask more domainspecific tokens, as compared to the state of the art (sota) @xcite , while mitigating spuriously masked tokens; ii) domain counterfactuals (d-con) from our model outperform the sota on binary and multi-label classification tasks, under unsupervised domain adaptation (uda) and adversarial domain adaptation (ada) settings, under almost all domain shifts. we denote the domain-agnostic information by x and the domain-specific text to be a specific fusion of x and d. we define a fusion function i, such that, x d := i(x; d). an inverse mapping i -1 makes it feasible to efficiently disentangle and extract the domain-generic information x. therefore, the task of domain counterfactual generation aims to find a function that essentially disentangles the domain-generic information from x d to obtain x, followed by mapping x to a domain-specific text x d ′ . thus, an ideal f @xmath0 overview. the proposed approach consists of two phases: • domain obfuscation: this phase masks a text x d , such that, it becomes void of source-domainspecific information. to this end, we propose a three-step masking approach. ## intrinsic evaluation conditional generation is difficult to automatically evaluate. therefore, we asked three ph.d. students trained in natural language processing to manually evaluate the dataset considering the following evaluation measures: (1) domain relevance ## models and baselines we tested on three types of models 5 : (a) baseline models (b) variants for each step in our masking approach c) an upper-bound model to approximate the best performance using domain counterfactuals (d-con) on the downstream tasks. unless otherwise stated, all the domain classifiers and sentiment classifiers use the same model, based on a pretrained roberta-base model, and all generation models are based on a pretrained t5-base. baseline da models. we experiment with four baselines: (1) no-domain-adaptation (noda), the model only trained on available training data from the source domain; (2) random-masking random-reconstruction (rmrr) @xcite , randomly masks token from the input example and then fills the masks by masked language modeling head. 3) perl (ben-david et al., 2020), a sota for the uda setup. 4) docogen: use the same model and training strategy as we do, but only use the first step of our masking procedure. ablations. we consider three types of ablations in our masking procedure. first, we show ablations for step 2 in our procedure. (5) 2-attention-score-masking (2-asm) where we use attention score rather than attention norm to perform oot masking step, (6) 3wo where we show the performance of unmasking by following word order in the sentence. and finally, we show (7) noinit, where we remove the heuristic masking step, and only use oot masking with unmasking. (8) no-unmask, 5 experiment 3 is not applicable to the ada setting due to lack of labeled data. ## results and discussions tables 3 and 4 present sentiment classification accuracy results for the 12 uda and 8 ada setups respectively. results comparison. for sentiment classification, our method remask, outperforms all baseline models in 10 of 12 uda setups and in 6 of 8 ada setups, gaining an average of 2% and 1.4% over the baseline masking approach in uda and ada settings, respectively. furthermore, in two ada setups on sentiment classification, our model outperforms the oracle-gen scores which suggests our model can effectively remove domain-specific information in these domain pairs. for intent prediction foot_4 , our model outperforms four out of six setups, reaching an average gain of 0.8% across domains. moreover, noinit result demonstrates that our model performance degrades if there is no initial mask provided by step 1. one key observation on the masking process is that step 2 of our method fails to identify some domain-specific tokens in the presence of too many such tokens in the text. we suspect that the thresholding attention norm is not robust enough when the attention norm might be thinly spread among many domain-specific tokens. on the other hand, remask outperforms doco-gen by 0.8% on the mantis dataset under the uda setup. this further confirms the efficacy of our method. we found that our remask have less performance improvement for domains where the domain specific words are distinctive like ub and have higher performance boost in domains that have more complex text, e.g. el and ph. ## conclusion this work empirically shows that the addition of attention norm-based masking leads to additional domain-token associations missed by the sota masking strategy. furthermore, our iterative unmasking approach leads to the removal of spurious domain-token associations, resulting in improved domain obfuscation. these two innovations collectively allow better domain transfer on sentiment and intent classifications tasks.