Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023)

Abstract: Machine learning (ML) systems in natural language processing (NLP) face significant challenges in generalizing to out-of-distribution (OOD) data, where the test distribution differs from the training data distribution. This poses important questions about the robustness of NLP models and their high accuracy, which may be artificially inflated due to their underlying sensitivity to systematic biases. Despite these challenges, there is a lack of comprehensive surveys on the generalization challenge from an OOD perspective in natural language understanding. Therefore, this paper aims to fill this gap by presenting the first comprehensive review of recent progress, methods, and evaluations on this topic. We further discuss the challenges involved and potential future research directions. By providing convenient access to existing work, we hope this survey will encourage future research in this area.

## introduction pre-trained language models (plms) @xcite @xcite have revolutionized natural language processing (nlp) and enabled remarkable advances in large-scale language models (llms) @xcite @xcite . despite substantial progress in developing accurate models in several natural language understanding tasks, including sentiment analysis @xcite @xcite @xcite , natural language inference @xcite , and machine reading comprehension @xcite , a major challenge persists -out-of-distribution (ood) generalization -which entails the ability of a model to 1 these authors contributed equally to this work. the importance of ood generalization in nlp cannot be overstated, as real-world data often exhibit diversity and unpredictability. numerous applications, such as sentiment analysis, document categorization, and spam detection @xcite , necessitate models capable of adapting to novel and unforeseen data distributions. while machine learning models generally demonstrate strong in-distribution performance, their performance frequently deteriorates when confronted with ood instances, underscoring the need for effective strategies that facilitate generalization beyond the training distribution. although research on ood generalization in nlp is emerging, it is not on the scale of other tasks like computer vision @xcite and time series @xcite . furthermore, most related surveys in nlp focus on measuring and improving model robustness against adversarial attacks @xcite , or providing causal explanations @xcite . among them, @xcite is the most relevant review to this paper, but their work does not differentiate between data-level variance and shortcut features and also not discuss llms. to address these limitations, this survey provides an extensive examination of the existing literature on ood generalization in nlp, covering a diverse array of techniques and approaches. we focus on two perspectives of ood generalization: the data distribution, which is model-independent and the feature distribution, which is model-oriented. ad-ditionally, we discuss the evaluation metrics and benchmarks employed to assess the effectiveness of these techniques, as well as the limitations and drawbacks of current methodologies. throughout this survey, we trace the evolution of ood generalization techniques in natural language processing, from the early approaches based on traditional machine learning algorithms to more recent advancements driven by deep learning architectures, also including the discussion of the most recent emergent abilities of llms. we identify the key innovations and breakthroughs that have shaped the field, while also highlighting areas where progress has been relatively slow or incremental. our analysis emphasizes the interconnected nature of these advancements and the importance of driving fundamental research in the generalization problem towards unforeseen data distributions. in addition, this survey aims to identify open challenges and future directions for ood generalization in nlp, especially for llms. we discuss the limitations of current techniques, potential avenues for improving model robustness and adaptability, and emerging research trends that may contribute to the development of more effective ood generalization strategies. the remainder of this survey is organized as follows: we formalize the scope of ood generalization in section 2. then, we propose a novel taxonomy towards ood robustness and review existing methodologies developed for addressing ood issues in section 3. in particular, we identify two salient aspects of ood generalization, namely data variance and shortcut features. we outline two representative application scenarios in section 4, namely deployment in high-stake domains and social bias. we also introduce the methods for improving the ood robustness in section 5 before discussing the redefinition of ood in the era of large language models. ## the scope of ood generalization denote a set of labeled data as @xmath0, y i )} n @xmath1, where an input @xmath2, output @xmath3, and n is the number of datasets. a training dataset d @xmath4, y train )} is generated by sampling from d with distribution p train , and the test dataset d @xmath5, y test )} is sampled from d with distribution p test . out-of-distribution (ood) refers to the circumstance when p train ≠ p test . in the context of text classification, let x be the set of all possible documents, y be the set of all possible labels, and d be a training distribution defined on x × y. suppose the true target distribution is p x ,y , which is close to but not identical to d with p x ,y ≠ d. when we encounter a document that is drawn from a distribution q x that is significantly different from p x , we refer to it as an out-of-distribution (ood) sample. an ood sample may have a vocabulary or language not presenting in p x . a text classification model f ∶ @xmath6. the ood detection function can be derived from a probabilistic perspective using bayesian inference. in this case, we can estimate the posterior probability of a document being ood given its bag-of-words features through bayesian model averaging: the conditional ood probability of model f θ on input x given the parameter θ is defined as: as can be seen from the above equations, ood can be perceived in terms of both the data-and model-levels, robust models can be more resistant to data variances. the ood detection function can be defined as a threshold on the posterior ood probability: ## taxonomy of out-of-distribution problems we classify ood problems into two perspectives, as depicted in figure 1 , namely data and features. data variance encompasses the domain generalization problem, while "shortcut features" represent a range of issues typically caused by shortcut learning, which cannot be avoided from inductive reasoning. ## application scenarios we highlight the importance of ood generalization in two real-world application scenarios, in which low ood robustness may lead to serious consequences. ## methods existing work to address ood issues in nlp can be categorized into three groups: data augmentation (sec.5.1), model-level control (sec.5.2), and training approaches (sec.5.3) shown in fig. 2 . descriptions of current ood generalization methods categorized by tasks are introduced in the appendix. ## large language models large language models (llms) have attracted increasing attention in the field of artificial intelligence recently. however, as a crucial property towards artificial general intelligence (agi), the ood robustness is still under-explored @xcite . given its importance, we review the recent work on the ood generalization of llms. ood definition it is of imminent importance to reframe the ood definition in the era of llm dominance since the pre-trained corpora of llms are not publicly available. the absence of pre-trained corpus information makes it hard to define ood examples for llms in nlp. although providing an accurate and strict definition remains challenging for large foundation models, researchers make attempts to build label-sharing ood data for llms from two perspectives, namely, synthetic data, and distribution shift over time. synthetic data is generally defined as artificially annotated information generated by algorithms or simulations, which can be hand-crafted as challenging ood examples for llms. distribution shift over time refers to the idea of using real-world datasets collected after 2021 as ood test data, which is the latest data collection time of chatgpt @xcite . another type of ood data refers to the task of generalizing to unseen classes. for instance, in open-set label shift @xcite , the test data includes examples from novel classes not present in the training data, making it impossible for classical small models to predict correctly. llms such as chatgpt can alleviate this issue by using incontext learning, as evidenced by recent research @xcite . this means that llms can be used to improve robustness with minimal human intervention but they cannot fully solve this problem and open-set label shift remains challenging. ood detection previous research on ood detection has employed models to identify test examples that come from a different distribution @xcite . some of these approaches introduce new training objectives, such as using a contrastive objective @xcite . when the type of distribution shift is known, the model can be trained to exhibit uncertainty when presented with known ood examples @xcite . however, the distribution of sota llms, such as chatgpt and gpt-4 is hidden and cannot be inferred. very recently, conal @xcite provides an alternative for generating novel examples which simulate open-set shifts and has proven to be effective for ood detection. regarding language models (llms), the deepfake detectors aimed at distinguishing content generated by humans or llms is closely related to previous algorithms designed for ood detection @xcite . when it comes to deepfake detection, one intuitive approach is to employ statistical boundaries that differentiate linguistic patterns between human-written and machine-generated text @xcite . however, these statistical methods have a limitation: they assume access to the model's prediction distributions is possible, which hinders their application to models behind apis, such as chatgpt. an alternative paradigm involves training neural-based detectors @xcite , including the official implementation of openai (openai, 2023a). ood robustness previous studies have extensively examined various aspects of chatgpt in the domains of law @xcite , ethics @xcite and planning @xcite . however, limited attention has been given to its robustness @xcite against out-of-distribution (ood) inputs. evaluating ood robustness in a reliable manner poses a significant challenge due to the massive and unknown training data of llms. @xcite offers an initial investigation into the robustness of chatgpt by presenting ood results on flipkart and ddxplus. building on this work, @xcite delimit the robustness of llms in comparison to conventional models, with a focus on aligning the threat model to the realistic deployment of llms. additionally, @xcite measures llms' resilience to adversarial prompts using adversarial textual attacks on character, word, sentence, and semantic levels. collectively, these evaluations raise similar concerns regarding the limited robustness of chatgpt. among different attack levels, character-level attacks demonstrate higher robustness, while word-level attacks pose the greatest vulnerability. recently, we have noticed that several papers survey the robustness issue in large language models (llms) from differ-ent perspectives, including factuality @xcite , hallucination @xcite , and evaluation methods @xcite . ## future directions and conclusion we consider multiple promising directions for improving the ood robustness from four perspectives: (1) enhancing the learning of such salient causal features, either by the help of human guidance @xcite via human-in-the-loop or through psychologically inspired neural structures @xcite , can be worthy of consideration; (2) data-centric ai: both the selection of training data and the careful design of prompt learning have proven effective in domain generalization @xcite . in addition, the emerging ability of large-scale language models holds a huge potential for ood generalization, benefiting from the instruction tuning, which requires a high-quality data construction process; (3) alignment methods: this can be effectuated through the deployment of reinforcement learning algorithms, be it in an online or offline setting @xcite ; (4) neuro-symbolic modeling for nlp: purely neural models like chatgpt can possess incredibly powerful generalization abilities. while it is more-or-less accepted that purely neural models face challenges of reasoning beyond surface-level patterns. in order to avoid picking up spurious correlations, neuro-symbolic approaches are proposed to improve the models' ood robustness by combining the learning capabilities of neural networks with the expressive power of symbolic reasoning @xcite @xcite . this paper presents an ambitious attempt to categorize the challenges of ood generalization, focusing on both data and model levels. by undertaking this categorization, our aim is to shed light on the limitations of current methods, emphasize the crucial nature of ood robustness, and provide quick access to existing references for further exploration. in addition, we emphasize the ongoing significance of ood robustness in the era of large language models, emphasizing the need to address this aspect. we call upon researchers in the nlp community to delve deeper into the proper definition of ood in the context of large models and develop appropriate benchmark tests that accurately measure the ood generalization ability of llms.