Example-based Hypernetworks for Multi-source Adaptation to Unseen Domains (2023)

Abstract: As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains’ semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier’s weights. In an advanced version, the signature also enriches the input example’s representation. We evaluated our method across two tasks—sentiment classification and natural language inference—in 29 adaptation scenarios, where it outpaced established algorithms. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To the best of our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.

## introduction deep neural networks (dnns) have substantially improved natural language processing (nlp), reaching task performance levels that were considered beyond imagination until recently @xcite . however, this unprecedented performance typically depends on the assumption that the test data is drawn from the same underlying distribution as the training data. unfortunately, as text may stem from many origins, this assumption is often not met in practice. in such cases, the model faces an out-ofdistribution (ood) generalization scenario, which often yields significant performance degradation. to alleviate this difficulty, several ood generalization approaches proposed to use unlabeled data from the target distribution. for example, a prominent domain adaptation (da, @xcite ) setting is unsupervised domain adaptation (uda, @xcite ), where algorithms use labeled data from the source domain and unlabeled data from both the source and the target domains @xcite @xcite . in many real-world scenarios, however, it is impractical to expect training-time access to target domain data. this could happen, for example, when the target domain is unknown, when collecting data from the target domain is impractical or when the data from the target domain is confidential (e.g. in healthcare applications). in order to address this setting, three approaches were proposed. the first approach follows the idea of domain robustness, generalizing to unknown domains through optimization methods which favor robustness over specification @xcite @xcite @xcite . particularly, these approaches train the model to focus on domain-invariant features and overlook properties that are associated only with some specific source domains. in contrast, the second approach implements a domain expert for each source domain, hence keeping knowledge of each domain separately. in this mixtureof-experts (moe) approach @xcite @xcite , an expert is trained for each domain separately, and the predictions of these experts are aggregated through averaging or voting. to bridge the gap between these opposing approaches, a third intermediate approach has been recently proposed by @xcite . their pada algorithm, standing for a prompt-based autoregressive approach for adaptation to unseen domains, utilizes domain-invariant and domain-premise. homes not located on one of these roads must place a mail receptacle along the route traveled. hypothesis. other roads are far too rural to provide mail service to. ## domain adaptation with hypernetworks in this section, we present our hn-based modeling framework for domain adaptation. we present three models in increased order of complexity: we start by generating parameters only for the task classifier in a domain-based manner (hyper-dn), proceed to example-based classifier parametrization (hyper-drf) and, finally, introduce examplebased parametrization at both the classifier and the text representation levels (hyper-pada). throughout this section we use the running example of ## results our results clearly demonstrate the superiority of hyper-pada and the simpler hn-based models. specifically, hyper-pada outperforms all baseline models (i.e. models that do not involve hypernetwork modeling, denoted bellow as non-hn models) in 11 of 12 clcd settings, in 8 of 12 cd sentiment settings, and in all 5 cd mnli settings, with an average improvement of 2.7%, 3.9% and 3.4% over the best performing baseline in each of the settings, respectively. another impressive result is the gap between hyper-pada and the t5-noda model, which does not perform adaptation: hyper-pada outperforms this model by 9.5%, 8.4% and 14.8% in clcd and cd sentiment classification and cd mnli, respectively. ## discussion we presented a hypernetwork-based framework for example-based domain adaptation, designed for multi-source adaptation to unseen domains. our framework provides several novelties: (a) the application of hypernetworks to unsupervised domain adaptation and any domain adaptation in nlp; (b) the application of hypernetworks in example-based manner (which is novel at least in nlp, to the best of our knowledge); (c) the generation of examplebased classifier weights, based on a learned signature which embeds the input example in the semantic space spanned by the source domains; and (d) the integration of all the above with an example representation mechanism that is based on the learned signature. while the concept of drf signatures stems from @xcite , the aforementioned innovations are unique to our work. comprehensive experiments across 2 tasks, 4 languages, 8 domains, and 29 adaptation settings underline our framework's superiority over previous methodologies and the value of our modeling choices. ## limitations extending beyond sequence classification although our experimental setup is broad and extensive, the tasks we considered are limited to sentence-level classification tasks. however, there are many other nlp tasks that present challenging out-of-distribution scenarios. since it is not trivial to extend hns effectively to token-level classification or text generation, we would like to address such cases in future work. ultimately, our goal is to shape our methodology to the level that nlp technology becomes available to as many textual domains as possible, with minimum data annotation and collection efforts. utilizing large models our modeling solution consists of a large pretrained language model. while one could apply the same method using smaller models (available today), it might lead to an unsatisfying performance level compared to the ones reported in this work. sentence. this documentary is poorly produced, has terrible sound quality and stereotypical "life affirming" stories. there was nothing in here to support wal-mart, their business practices or their philosophy. domain. dvd. label. negative. drf signature. music: history, rock, sound, story hyper-drf and hyper-pada we have to construct a drf signature for each training example. our goal in this process is to match each training example with those drfs in its domain's drf set that are most representative of its semantics. we do this in an automatic manner. let w 1 , ...w n be the tokens of a sentence x from the domain s j . each drf r @xmath0: to summarize, we utilize this annotation only during training, as a training signal for the drf signature generator (in stage 1 of both hyper-drf and hyper-pada). tables 1 (main paper) and 4 provide mnli and sentiment classification examples and their drf signatures, as generated by hyper-pada and hyper-drf in a specific adaptation setup.