Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP (2023)

Abstract: When deploying machine learning systems to the wild, it is highly desirable for them to effectively leverage prior knowledge to the unfamiliar domain while also firing alarms to anomalous inputs. In order to address these requirements, Universal Domain Adaptation (UniDA) has emerged as a novel research area in computer vision, focusing on achieving both adaptation ability and robustness (i.e., the ability to detect out-of-distribution samples). While UniDA has led significant progress in computer vision, its application on language input still needs to be explored despite its feasibility. In this paper, we propose a comprehensive benchmark for natural language that offers thorough viewpoints of the model’s generalizability and robustness. Our benchmark encompasses multiple datasets with varying difficulty levels and characteristics, including temporal shifts and diverse domains. On top of our testbed, we validate existing UniDA methods from computer vision and state-of-the-art domain adaptation techniques from NLP literature, yielding valuable findings: We observe that UniDA methods originally designed for image input can be effectively transferred to the natural language domain while also underscoring the effect of adaptation difficulty in determining the model’s performance.

## introduction deep learning models demonstrate satisfactory performance when tested on data from the training distribution. however, real-world inputs encounter novel data ceaselessly that deviate from the trained distribution, commonly known as distributional shift. when confronted with such inputs, machine learning models frequently struggle to differentiate them from regular input. consequently, they face challenges in adapting their previously acquired knowledge to the new data distribution, resulting sorry, i cannot handle your request. sure. i'll open the window. whip me up a sandwich real quick? figure 1 : the model trained with formal language (source domain) will likely face spoken language (target domain) in the real world. the model is expected to properly handle such transferable input despite the distributional shift. (middle) at the same time, the model should discern unprocessable inputs (bottom) from the target domain. the aforementioned phenomenon represents a longstanding challenge within the machine learning community, wherein even recent cutting-edge language models @xcite @xcite do not serve as an exception to this predicament @xcite . in response to these challenges, existing literature proposes two distinct approaches. the first approach, known as domain adaptation (da) @xcite @xcite , endeavors to establish alignment between a new set of data from an unknown distribution and the model's prior knowl-edge distribution. the objective is to enhance the model's generalization capability and reduce performance drop springing from the distributional shift. in parallel, a distinct line of work, referred to as out-of-distribution (ood) detection @xcite @xcite , focuses on discerning inputs originating from dissimilar distributions. they opt to circumvent potential risks or disruptions arising from shifted inputs, thereby enriching system robustness and resilience. while both approaches offer unique advantages addressing specific distributional shifts, integrating their merits could substantially enhance robustness. in pursuit of this objective, a novel field called universal domain adaptation (unida) @xcite has emerged, aiming to harness the synergies of both ood detection and da when confronted with distributional shifts. unida leverages the best of the two worlds and offers comprehensive perspectives that integrate the merits of these two research areas. the essence of unida lies in measuring the uncertainty of the data from the shifted distribution precisely. then, we can enhance the model's transferability by distinguishing portions of low-uncertainty inputs that can be adequately handled with the current model's knowledge. simultaneously, we enrich the robustness of the model to ood inputs by discerning the remaining samples that cannot be processed normally. however, distinguishing between these inputs and properly processing them becomes increasingly challenging without explicit supervision. despite the versatility of unida, this topic has yet to be explored in the natural language processing (nlp) literature. as a cornerstone in enhancing reliability against distributional shifts in nlp, we introduce a testbed for evaluating the model's robustness in a holistic view. first, we construct various adaptation scenarios in nlp, utilizing an array of thoughtfully selected datasets. to discern the degree to which our proposed datasets incorporate the various degree of challenges in unida, we define two novel metrics: performance drop rate (pdr) and distinction difficulty score (dds). using these metrics, we verify that our testbed captures a broad spectrum of distributional shifts. finally, based on the suggested setting, we systematically compare several unida methods inherently designed for the task, against heuristic combinations of previous approaches for the parts of the problem, i.e., ood detection and da. our empirical results show that unida methods are fully transferable in the nlp domain and can robustly respond to various degrees of shift. moreover, we find out that the adaptation difficulty notably affects the performance of the methods. in certain circumstances, da methods display comparable or even better performance. we release our dataset, encouraging future research on unida in nlp to foster the development of more resilient and domain-specific strategies. foot_0 2 universal domain adaptation ## testbed design the primary objective of our research is to construct a comprehensive benchmark dataset that effectively captures the viewpoint of unida. to accomplish our objective, we attempt to create a diverse dataset that encompasses a range of difficulty levels and characteristics, such as domains, sentiment, or temporal change. these variations are the fundamental elements that can significantly influence the overall performance. specifically, we initially select datasets from multiple practical domains and approximate the adaptation difficulty by quantifying different shifts with our newly proposed metrics. in the following subsections, we provide an in-depth explanation of our dataset along with the analysis of our benchmarks. ## conclusion and future work in this study, we present a testbed for evaluating unida in the field of nlp. the testbed is designed to exhibit various levels of domain and category gaps through different datasets. two novel metrics, pdr and dds, were proposed which can measure the degree of domain and category gap, respectively. we assessed unida methods and the heuristic combination of cda and ood detection in our proposed testbed. experimental results show that unida methods, initially designed for the vision domain, can be effectively transferred to nlp. additionally, cda methods, which are not fully optimized in unida scenario, produce comparable results in certain circumstances. recent trends in nlp focus on large language models (llms) of their significant generalization abilities. however, the robustness of llms from the perspective of unida remains uncertain. as part of our future work, we assess the performance and the capabilities of llms from a unida viewpoint.