Counterfactual Augmentation for Multimodal Learning Under Presentation Bias (2023)

Abstract: In real-world machine learning systems, labels are often derived from user behaviors that the system wishes to encourage. Over time, new models must be trained as new training examples and features become available. However, feedback loops between users and models can bias future user behavior, inducing a *presentation bias* in the labels that compromises the ability to train new models. In this paper, we propose *counterfactual augmentation*, a novel causal method for correcting presentation bias using generated counterfactual labels. Our empirical evaluations demonstrate that counterfactual augmentation yields better downstream performance compared to both uncorrected models and existing bias-correction methods. Model analyses further indicate that the generated counterfactuals align closely with true counterfactuals in an oracle setting.

## introduction deployment of machine learning models is ubiquitous in the real world, ranging from web search ranking to movie recommendation. to ensure good performance, new models must be trained periodically, since new training examples may become available, and the types of features that are collected can evolve over time (e.g., from tabular to multimodal data). for user-facing models like recommenders, labels are often derived from user behaviors that the model wishes to encourage, and user-model interactions continuously produce new data that can be used for training models. modern nlp, for instance, relies heavily on models that learn from user feedback, not the least of which are chatgpt and other large language models that comprise the current state-of-the-art. in practice, however, feedback loops between the user and the model can influence future user * the majority of this work was conducted while this author was an intern at microsoft. figure 1 : an illustration of how presentation bias may arise from feedback loops (e.g., in a movie recommendation system). the top sequence depicts uncorrected presentation bias, while the bottom sequence demonstrates how our method, counterfactual augmentation, can correct presentation bias. presentation bias negatively affects the data distribution in two major ways. (1) bias amplification. new labels are dependent on the prior behavior of the model, so they may not reflect the user's true preferences. this bias will amplify as more training loops are completed on biased data. (2) label homogenization. as the model learns user behaviors, most users' responses to its recommendations will be positive, so variation in user feedback decreases. in this paper, we aim to correct the presentation bias resulting from feedback loops. we first propose that presentation bias arises due to the causal relationship between a model's recommendations and a user's behavior, which affects which labels are observed. users tend to interact with recommended items, so under presentation bias, we are more likely to observe labels for recommended items-while without presentation bias, users would interact with all items (or a random subset), so labels would be observed for the full distribution. we conclude that we can break the causal link behind presentation bias with a counterfactual question: how would users have reacted had they interacted with all items, contrary to reality? with this idea as our foundation, we introduce counterfactual augmentation (figure 1 ), a causal approach for reducing presentation bias using generated counterfactual labels. 1 because "true" counterfactuals are by definition unknown, counterfactual augmentation leverages the causal relationship between the model's behavior and the user's behavior to generate realistic counterfactual labels. we generate counterfactuals for the labels that are unobserved due to presentation bias, then augment the observed labels with the generated ones. intuitively, this supplies labels over the full data distribution, yielding a bias-corrected dataset. we evaluate our method on predictive tasks in language and multimodal settings that reflect realworld presentation bias. we consider data with evolving feature spaces, where over time the features transition from simpler features to richer language or multimodal ones. 2 in our experiments, we demonstrate that counterfactual augmentation effectively corrects presentation bias when training predictive models, outperforming both uncorrected w t , and y t denote simple features, rich features, and labels at time t (no subscript for @xmath0, while r t is a model (e.g., a recommender) trained over the input features and labels. a indicates which items a user interacts with. ## problem statement we formalize the problem of presentation bias in machine learning systems in causal terms (figure 2 ). these systems usually consume both simple features, such as metadata, and rich features, such as text or images, training on user interactions with different items to produce recommendations. let t be a time index, and let x t denote the simple features defined over the feature space x. similarly, let w t denote rich features defined over the feature space w. we denote true user item preferences as y @xmath1, we can think of these as binary recommendations). finally, let a t be an indicator of which items the user interacts with. due to feature evolution, only x t is observed at earlier time points, while later both x t and w t are observed. without loss of generality, assume that the feature evolution occurs in the first two time points @xmath2. for ease of notation, we do not include time index subscripts when @xmath3. at @xmath4, a predictive model r 0 is trained on an observed feature set x 0 and labels y 0 . this model makes predictions about user preferences for unseen items and recommends items to the user. these recommendations influence a-which of those items the user subsequently interacts withbecause users are much more likely to interact with recommended items, such that p (@xmath5. in turn, this induces a presentation bias in the distribution of observed y , the user's measured preferences at @xmath6. due to the presentation bias, there is a very high probability of observing y when r 0 = 1 and a very low probability of observing y when r 0 = 0. at @xmath7, a full set of simple and rich features (x, w ) is observed due to feature evolution. however, because the distribution of y has been influenced by r 0 , a second model r trained on x, w , and y will not correctly learn user preferences. example. consider a system that must categorize a user's emails as important and unimportant. x is email metadata, and w is the text of the email. y is an indicator of whether the user interacted with the email positively (e.g., replied) or negatively (e.g., reported spam). r 0 is a classifier trained on x 0 and y 0 to label emails as important or unimportant. users preferentially interact with important emails, so emails with r 0 = 1 have a much higher chance of user interaction (i.e., @xmath8, inducing a bias in y that depends on r 0 . after r 0 is trained, the system's administrators want to train a new, improved model r using both x and w . however, the bias in y will affect the ability to train r. ## methods to eliminate presentation bias, we notice that we must block the causal path between r 0 and y so that r 0 no longer influences which y are observed. because these two variables are linked by the mediator a, we can block the path by controlling for a. to do so, we define the counterfactual y @xmath9, the value y would have taken had @xmath10. ## experiments we conduct empirical evaluations to assess how well counterfactual augmentation corrects presentation bias, with the aim of improving downstream performance. we evaluate on predictive machine learning tasks, which reflect real-world models' goals of predicting user behavior, and in multiple data settings, both synthetic and real-world. to facilitate detailed analysis of our models, we introduce a procedure for inducing realistic presentation bias in unbiased datasets. all data and code for our experiments will be publicly released. ## related work presentation bias may be considered a type of selection bias, in which the sampling distribution differs from the population distribution. selection bias is a core challenge of observational causal inference, where the causal effect of a treatment a on an outcome y is estimated not from a randomized trial but from observed data. since the treatment assignment mechanism is not random, it must be accounted for during estimation. one common method for addressing selection bias in causal inference is inverse propensity weighting (ipw) @xcite . at a high level, ipw up-weights samples corresponding to treatment conditions that are unlikely to be observed and down-weights samples corresponding to treatment conditions that are likely to be observed, such that all treatment conditions appear to be equally likely over the data distribution. this blocks the causal path between the treatment assignment and the outcome. ipw for presentation bias correction. using this principle, a number of works propose an inverse propensity weighted empirical loss function that can be used to reduce the effects of presentation bias when training a model on biased data @xcite @xcite . several works also engage with ipw in more complex ways. @xcite address a longitudinal bias setting and propose an algorithm that maximizes the desired outcome at each time step using an ipw-based estimator. @xcite introduce dragonnet, a fully-connected multi-head neural network that jointly predicts the treatment and the outcome, simultaneously yielding both a propensity score estimate and a predicted outcome. task-based presentation bias correction. because presentation bias can appear in many task settings, there exist a number of task-specific approaches for reducing presentation bias. in information retrieval, for example, unbiased learners of click data @xcite and propensity-weighted rank metrics @xcite have been proposed, while in the recommender literature, methods have been developed for the matrix factorization setting @xcite . however, the task-specific nature of these methods limits their generalizability compared to counterfactual augmentation. estimating counterfactuals. the inability to know an individual's counterfactual is a central challenge of causal inference. however, recent works in the deep learning literature have made large inroads toward estimating individual treatment effects @xcite @xcite , which is an adjacent task to estimating individual counterfactuals. we draw upon this body of work as a basis for obtaining high-quality counterfactuals. ## conclusion in this paper, we introduced counterfactual augmentation, a causal method for correcting presentation bias using generated counterfactuals. we described the causal mechanism behind presentation bias in real-world machine learning systems that rely on user feedback, and we explained the causal reasoning behind counterfactual augmentation. we presented empirical evaluations using counterfactual augmentation to reduce presentation bias, and we found that our approach significantly outperforms existing methods. finally, we conducted a model analysis to explore why counterfactual augmentation is effective in addressing presentation bias. given the prevalence of presentation bias in real-world deployments of machine learning models, our findings suggest that counterfactual augmentation has the potential to improve the quality of user-facing machine learning models across many types of applications.