D o C o G en: D omain Counterfactual Generation for Low Resource Domain Adaptation (2022)

Abstract: Natural language processing (NLP) algorithms have become very successful, but they still struggle when applied to out-of-distribution examples. In this paper we propose a controllable generation approach in order to deal with this domain adaptation (DA) challenge. Given an input text example, our DoCoGen algorithm generates a domain-counterfactual textual example (D-con) - that is similar to the original in all aspects, including the task label, but its domain is changed to a desired one. Importantly, DoCoGen is trained using only unlabeled examples from multiple domains - no NLP task labels or parallel pairs of textual examples and their domain-counterfactuals are required. We show that DoCoGen can generate coherent counterfactuals consisting of multiple sentences. We use the D-cons generated by DoCoGen to augment a sentiment classifier and a multi-label intent classifier in 20 and 78 DA setups, respectively, where source-domain labeled data is scarce. Our model outperforms strong baselines and improves the accuracy of a state-of-the-art unsupervised DA algorithm.

## introduction natural language processing (nlp) algorithms are constantly improving and reaching significant milestones @xcite @xcite . however, such algorithms rely on the availability of sufficient labeled data and the assumption that the training and test sets are drawn from the same underlying distribution. unfortunately, these assumptions do not hold in many cases due to the costly and labor-intensive data labeling process and since text may originate from many different domains. as generalization in low resource regimes and beyond the training distribution are still fundamental nlp challenges, nlp algorithms significantly degrade when applied to such scenarios. domain adaptation (da) is an established field of research in nlp @xcite @xcite ) that attempts to explicitly address generalization beyond the training distribution ( §2). da algorithms are trained on annotated data from source domains to be effectively applied in various target domains. indeed, da algorithms have been developed for multiple nlp tasks throughout the last two decades @xcite @xcite @xcite @xcite . a natural alternative to costly human annotation would be to automatically generate labeled examples for model training. doing so may expose the model to additional training examples and better represent the data distribution within and outside the annotated source domains. unfortunately, generating labeled textual data is challenging @xcite , especially when the available labeled data is scarce. indeed, labeled data generation has hardly been applied to da ( §2). to allow da through labeled data generation, we present docogen, an algorithm that generates domain-counterfactual textual examples (d-cons). in order to do that, docogen intervenes on the domain-specific terms of its input example, replacing them with terms that are relevant for its target domain while keeping all other properties fixed, including the task label. consider the task of sentiment classification (top example in docogen is a controllable generation algo-rithm @xcite ) that is trained using a novel unsupervised sentence reconstruction objective. importantly, it does not require task-annotated data, or parallel pairs of sentences and their d-cons. a key component of docogen is the domain orientation vector, which guides the model to generate the new text in the desired domain. the parameters of the orientation vectors are learned during the unsupervised training process, allowing the generation model to share information among the various domains it is exposed to. we focus on two low resource scenarios: unsupervised domain adaptation (uda) and any domain adaptation @xcite ), with only a handful of labeled examples available from a single source domain. in both uda and ada the model is exposed to limited labeled source domain data and to unlabeled data from several domains. however, in uda the unlabeled domains contain the future target domain to which the model will be applied, while in ada the model has no access to the target domain during training. to cope with these extreme conditions, we use docogen to enrich the source labeled data with d-cons from the unlabeled domains. by introducing labeled d-cons from various domains, we hope to provide the model with a training signal that is less affected by spurious correlations: correlations between features and the task label which do not hold out-of-domain (ood) @xcite . after a brief evaluation of the intrinsic quality of the d-cons generated by docogen, we evaluate our complete da pipeline. we focus on two tasks: binary sentiment classification of reviews and multi-label intent prediction in informationseeking conversations. in both tasks, we follow the uda and ada scenarios, for a total of 12 and 8 sentiment setups, respectively, as well as 30 uda and 48 ada intent prediction setups. our results demonstrate the superiority of docogen over strong da and textual-data augmentation algorithms. finally, combining docogen with perl @xcite , a sota uda model, yields new sota da accuracy and stability. ## related work we first describe research in our da setups: uda and ada. we then continue with the study of counterfactual-based data augmentation, and, finally, we describe research on counterfactual generation methods. ## domain-counterfactual examples in this section, we formally define the concept of domain-counterfactual textual examples (d-cons) and discuss the motivation behind them. definition x is a domain-counterfactual example (d-con) of x if it is a coherent human-like text that is a result of intervening on the domain of x and changing it to another domain, while holding everything else equal. particularly, we would like the task label of x and x to be identical. formally, given an example (x, y) ∼ d and a destination domain d , the goal of d-con generation is to generate x ∼ p d (x|@xmath0, where d is the domain counterfactual operator. in this work, given a labeled source example x we aim to generate coherent human-like d-cons from the unlabeled domains (see §1). we propose a d-con generation algorithm, docogen, consisting of two components. the first involves masking domain specific terms of the given example, yielding m(x). the second is a controllable generation model g which takes as input m(x) and a domain orientation vector v . this vector specifies the destination domain d , controlling the semantics of the generated d-con. formally: motivation the nlp community has recently become increasingly concerned with spurious correlations @xcite @xcite . in the case of da, spurious correlations may be defined as correlations between x and y which are relevant only to a specific domain or in a certain sample of labeled examples. such correlations may make a predictor f : @xmath1. using counterfactuals w.r.t. a specific variable allows us to both estimate its effect on our predictor @xcite or alleviate its impact on it @xcite . we focus on the latter, automatically generating d-cons by intervening on the domain variable d. adding these d-cons to the training set of a predictor should reduce its reliance on domainspecific information and spurious correlations. from a da perspective, enriching the training data with d-cons is motivated by pivot features ( §2), which are frequent in multiple domains and are prominent for the task. d-cons preserve language patterns, such as pivots, which are frequent in multiple domains. consider the middle example in ## docogen: domain counterfactual generation we propose a corrupt-and-reconstruct approach for generating d-cons from given source domain examples (figure 1 ). we next extend on these two steps, and describe our filtering mechanism used to disqualify low quality d-cons. ## intrinsic evaluation we next assess docogen in terms of its generated d-cons, ensuring they: (i) belong to the correct domain and label (1, 2), and (ii) are fluent (3, 4). to this end, we collected 20 original reviews, equally distributed among four domains (the a, d, e, and k domains, see §6). we then applied docogen to generate 60 d-cons, 3 for each of the original reviews (see §6 for the docogen training setup). finally, we trained the vae model of @xcite on labeled data (all the labeled data of the a, d, e, and k domains) and applied it to generate five reviews from each of the above four domains, with the same number of positive and negative reviews as in the set of original reviews. we then conducted a crowd-sourcing experiment where five nearly native english speakers rated each example, considering the following evaluation measures: (1) domain relevance (d.rel) -whether the topic of the generated text is related to its destination domain; (2) label preservation (l.pres)what is the label of the generated example (and we report whether the answer was identical to the desired label); (3) linguistic acceptability (accpt)how logical and grammatical the example is (on a 1-5 scale); and ( 4 ) word error rate (wer) -what is 2 §b.3 presents the % of masked tokens in our experiments. ↑d.rel ↑l.pres ↑accpt ↓wer vae 90.0 46.0 2.11 0.54 docogen 93.0 80.0 4.01 0.17 original reviews 99.0 88.0 4.73 0.10 the minimum number of word substitutions, deletions, and insertions that have to be performed to make the example logical and grammatical. 3 in this subsection we describe our tasks and datasets, as well as the two da setups which are the focus of this work. a full description of the number of samples in each dataset is found in sentiment classification we follow a large body of prior da work, focusing on the task of binary sentiment classification. specifically, our experiments include six different domains: the four legacy product review domains @xcite -books (b), dvds (d), electronic items (e) and kitchen appliances (k); the challenging airline review dataset (a) @xcite ; and the restaurant (r) domain obtained from the yelp dataset challenge @xcite . the focus of this work is on low resource da, and thus we randomly sample 100 labeled examples to form the training set for the following domains: a, d, e, and k. as described in §2, we explore two da setups, uda and ada. for uda, where the model has access to unlabeled target domain data, we experiment with 12 cross-domain setups, including the following domains: a, d, e, and k. for ada, where unlabeled data from the target domain is not within reach, we experiment with a total of 8 setups, including b and r as target domains, and a, d, e, and k as source domains. our reported accuracy scores are averaged across 25 different seeds and randomly sampled training and development sets. multi-label intent prediction our second task is multi-label intent prediction of utterances from information-seeking conversations. we use the multi-domain mantis dataset @xcite , consisting of diverse conversations from the question-answering stack exchange portal. the authors provide manually annotated user intent utterances, with eight possible intent labels, such as information request, potential answer and greetings. since we focus on low resource scenarios, we use only the five most common labels, as the frequency of the other three labels is less than 5%, and in some domains they are completely missing. the mantis dataset consists of 14 domains: apple (ap), dba (db), electronics (el), physics (ph), statistics (st), askubuntu (ub); diy (di), english (en), gaming (ga), gis (gi), sci-fi (sc), security (se), travel (tr) and worldbuilding (wo). we use the first 6 domains as unlabeled domains, randomly sampling train, development and test sets for each. the remaining 8 domains are used as target domains in the ada setup, resulting in 30 uda (6 × 5) and 48 ada (6 × 8) setups. following @xcite , we use the (macro) f1-score to measure classifier performances, and, like in the sentiment classification task, our reported results are averaged across 25 different seeds and randomly sampled training sets. ## results tables 3 and 4 present sentiment classification accuracy results for the 12 uda and 8 ada setups, respectively. ## conclusions we presented docogen, a corrupt-and-reconstruct approach for generating domain-counterfactuals (d-cons) and apply it as a data augmentation method in low-resource da. we hypothesized that d-cons may mitigate the reliance on domainspecific features and on spurious correlations and help generalize out of domain. our augmentation strategy yields robust models that outperform strong baselines across many lowresource da setups. in future work we would like to further improve the controllable generation quality of docogen, potentially extending it to control for multiple attributes. moreover, we would like our methodology to address additional nlp tasks and da setups.