Principles from Clinical Research for NLP Model Generalization (2024)

Abstract: The NLP community typically relies on performance of a model on a held-out test set to assess generalization. Performance drops observed in datasets outside of official test sets are generally attributed to “out-of-distribution” effects. Here, we explore the foundations of generalizability and study the factors that affect it, articulating lessons from clinical studies. In clinical research, generalizability is an act of reasoning that depends on (a) *internal validity* of experiments to ensure controlled measurement of cause and effect, and (b) *external validity* or transportability of the results to the wider population. We demonstrate how learning spurious correlations, such as the distance between entities in relation extraction tasks, can affect a model’s internal validity and in turn adversely impact generalization. We, therefore, present the need to ensure internal validity when building machine learning models in NLP. Our recommendations also apply to generative large language models, as they are known to be sensitive to even minor semantic preserving alterations. We also propose adapting the idea of *matching* in randomized controlled trials and observational studies to NLP evaluation to measure causation.

## introduction what factors lead to poor generalizability of models? to understand causes of non-generalization, the notion of generalizability needs to be first clearly defined. one definition of generalizability, in the context of general machine learning, is the ability of a model to perform well on data unseen during training, but drawn from the same distribution or population @xcite . data unseen during training clearly refers to data that is not part of training data, and is arguably uncontroversial to support sound evaluation. however, the requirement of data drawn from the same distribution or population warrants more scrutiny. it is important to consider the impact of data distribution on generalization, yet this is very challenging for natural language data. in statistics, the concept of distribution indicates "the pattern of variation in a variable or set of variables in the multivariate case", and thus describes the frequency of values of an observed variable @xcite . distribution or frequency of observed variable values can be challenging notions to meaningfully adapt to high dimensional, multivariate, and high variability text data. currently, no formal definition of in-distribution or out-of-distribution (ood) texts, or how to detect them, exists for nlp @xcite . where distributions are considered, simple surface linguistic characteristics or lexical-level distributions are emphasized @xcite . despite the lack of a comprehensive formal definition of ood in existing nlp literature, ood has become the most commonly cited reason for generalization failure when a model performs poorly outside an official test set. due to the black-box nature of deep learning models, it is increasingly difficult to demonstrate if a model has established a robust decision-making process that is generalizable to unseen data. as a consequence, generalization failures are typically ascribed to external factors, i.e. those extraneous to model development practices, and primarily to shifts in data distribution, or ood. recent studies have emerged showing that the robustness of a model can be undermined by inadvertent errors made during development of a model. for example, it has been shown that data leakage from training into test splits can lead to inflated test results @xcite . other works have pointed out spurious correlations in various benchmark datasets that may be leveraged by deep learning models to achieve inflated performances on test sets, while having poor generalization capability to real-world settings @xcite @xcite . in this paper, we hope to draw attention to the causes of nlp model generalization failures, especially those internal factors that are part of the model development process (e.g. preparation of training data). we argue that the external validity of a model should only be examined if the internal validity of the model is established. we start with a case study, showing how generalization failures can be caused by internal factors -the model has learned surface patterns in training data. we then propose that a pragmatic notion of model generalizability in the nlp domain can be established through borrowing and adapting practices from a domain seemingly far afield -clinical research. the contribution of this paper is two-fold. first, we show how ood may not be the sole cause of generalization failures, via a relation extraction task, highlighting the need for intrinsic investigation of why models fail. second, we propose to categorize the causes of generalization failures in nlp models, drawing inspiration from clinical studies. our work provides guidance on how to more systematically analyze generalization failures and adapt the principles behind randomized controlled trials for nlp model evaluation. ## relation extraction case study through a relation extraction case study over two data sets, we demonstrate that poor generalization in real-world application can result from ineffective modelling, i.e. learning of superficial surface patterns, rather than data distribution shift. for completeness, we also study a popular benchmark dataset on natural language inference (nli) task. ## foundations of generalizability the core idea behind generalizability is that the conclusions drawn, or a model inferred, from a sample can be applied to a wider population. in this section, we discuss some of the foundations of generalizability from clinical studies and why generalization failures cannot be solely attributed to out-of-distribution factors. identifying the cause of generalization failures requires several components to be well-defined ahead of the experiments, which are discussed in detail in this section. ## related works there are several reports of spurious correlations in sota models and benchmark datasets mccoy et al. (2019); gururangan et al. (2018); shinoda et al. (2022) , as discussed previously in this paper. @xcite attempt to formalize the definition of which features can be deemed spurious. approaches to detect spurious features without apriori handcrafting them have also been studied, e.g. utama et al. (2020), relying on the observation that pre-trained models exploit simple patterns in the early stage of the training phase. training a model to be robust against spurious correlations is also an active area of research, e.g. @xcite find that multitask learning improves generalization. datadriven approaches such as the use of h-adversarial training samples @xcite , appropriate sample selection @xcite , or robust loss function @xcite can improve model robustness to spurious correlations. causal deep learning and inference can also assist in ensuring causality @xcite . ## conclusion generalization is an act of reasoning that involves drawing broad inferences from specific observations @xcite . while generalization of machine learning models in nlp is a complex topic, clinical research offers guiding principles on how to understand generalization, including precise population definitions and contrasting controls. there are various points of failure in nlp modelling that can produce results that may not be generalizable. out-of-distribution (ood) effects may be an overly simplistic explanation for generalization failures. attributing causation to generalization failures requires detailed investigation into some potential pitfalls affecting internal validity. ensuring internal validity requires at least an effective test set that controls for spurious correlations in the training data. more careful construction of evaluation frameworks will ensure appropriate inferences about the relationship between cause (model's linguistic capabilities) and effect (performance) from experimental results. train ground truth @xmath0: decision tree (nb-t) fit in train ground truth fit 2306 test ground truth @xmath1: decision tree (nb-t) fit in test ground truth fit