Anticipating Performativity by Predicting from Predictions
Abstract
Predictions about people, such as their expected educational achievement or their credit risk, can be performative and shape the outcome that they are designed to predict.
Understanding the causal effect of predictions on the eventual outcomes is crucial for foreseeing the implications of future predictive models and selecting which models to deploy.
However, this causal estimation task poses unique challenges: model predictions are usually deterministic functions of input features and highly correlated with outcomes. This can make the causal effect of predictions on outcomes impossible to disentangle from the direct effect of the covariates.
We study this problem through the lens of causal identiﬁability.
Despite the hardness of this problem in full generality, we highlight three natural scenarios where the causal effect of predictions can be identiﬁed from observational data: randomization in predictions, overparameterization of the predictive model deployed during data collection, and discrete prediction outputs.
Empirically we show that given our identiﬁability conditions hold, standard variants of supervised learning that predict from predictions by treating the prediction as an input feature can ﬁnd transferable functional relationships that allow for conclusions about newly deployed predictive models.
These positive results fundamentally rely on model predictions being recorded during data collection , bringing forward the importance of rethinking standard data collection practices to enable progress towards a better understanding of social outcomes and performative feedback loops.
Introduction
Predictions can impact sentiments, alter expectations, inform actions, and thus change the course of events.
Through their inﬂuence on people, predictions have the potential to change the regularities in the population they seek to describe and understand.
This insight underlies the theories of performativity [38] and reﬂexivity [62] that play an important role in modern economics and ﬁnance.
Recently, Perdomo et al.
[51] pointed out that the social theory of performativity has important implications for machine learning theory and practice.
Prevailing approaches to supervised learning assume that features X and labels Y are sampled jointly from a ﬁxed underlying data distribution that is unaffected by attempts to predict Y from X .
Performativity questions this assumption and suggests that the deployment of a predictive model can disrupt the relationship between X and Y .
Hence, changes to the predictive model can induce shifts in the data distribution.
For example, consider a lender with a predictive model for risk of default – performativity could arise if individuals who are predicted as likely to default are given higher interest loans, which make default even more likely [41], akin to a self-fulﬁlling prophecy.
In turn, a different predictive model that predicts smaller risk and suggests offering more low-interest loans could cause some individuals who previously looked risky to be able to pay the loans back, which would appear as a shift in the relationship between features X and loan repayment outcomes Y .
This performative nature of predictions poses a challenge to using historical data to predict the outcomes that will arise under the deployment of future models.
1.1 Our work
In this work, we aim to understand under what conditions observational data is sufﬁcient to identify the performative effects of predictions.
Only when causal identiﬁability is established can we rely on data-driven strategies to anticipate performativity and reason about the downstream consequences of deploying new models.
Towards this goal, we focus on a subclass of performative prediction problems in this paper where performative effects of predictions solely surface as a shift in the outcome variable, and the distribution over covariates X is unaffected by the prediction ˆ Y .
Our goal is to identify the expected counterfactual outcome
[Equation]
Understanding the causal mechanism M Y is crucial for model evaluation, as well as model optimization.
In particular, it allows for ofﬂine evaluation of the potential outcome Y of an individual X subject to a predictive model f new with the prediction ˆ Y =
f new ( X ) before actually deploying it.
The need for observing predictions.
We start by illustrating the hardness of performativity-agnostic learning by relating performative prediction to a concept shift problem.
Using the speciﬁcs of the performative shift, we establish a lower bound on the extrapolation error of predicting Y from X under the deployment of a new model f new that is different from the model f train deployed during data collection.
In particular, the extrapolation error grows with the distance between the prediction functions of the two models and the strength of performativity.
This lower bound on the extrapolation error demonstrates the necessity to take performativity into account for reliably predicting Y .
Predicting from predictions.
We then explore the feasibility of learning performative effects when the training data recorded the predictions and training data samples ( X, Y, ˆ Y ) are available.
As an identiﬁcation strategy for learning M Y , we focus on building a meta machine learning model that predicts Y for an individual with features X , subjected to a prediction ˆ Y .
We term this data-driven strategy predicting from predictions ; it treats the predictions as an input to the meta machine learning model.
The meta model seeks to answer “what would the outcome be if we were to deploy a different prediction model?”
Crucially, this “what if” question is causal in nature; it aims to understand the potential outcome under an intervention which is different from merely estimating the outcome variable in previously seen data.
Whether such a transferable model is learnable depends on whether the training data provides causal identiﬁability [49] Only after causal identiﬁability is established can we rely on observational data to select and design optimal prediction models under performativity.
Establishing identiﬁability.
For our main technical results, we ﬁrst show that, in general, observing ˆ
Y is not sufﬁcient for identifying the causal effects of predictions.
In particular, if the training data was collected under the deployment of a deterministic prediction function, the mechanism M Y can not be uniquely identiﬁed.
The reason is a lack of coverage in the training data as X and ˆ Y are deterministically bound.
Next, we establish several conditions under which observing ˆ Y is sufﬁcient for identifying M Y .
The ﬁrst condition exploits the presence of randomness in the prediction.
This randomness could be purposely built into the prediction for individual fairness, differential privacy, or other considerations.
The second condition exploits the property that predictive models are often over-parameterized, which leads to incongruence in functional complexity between different causal paths, enabling the effects of predictions to be separated from other variables’ effects.
The third condition takes advantage of discreteness in predictions such that performative effects can be disentangled from the continuous relationship between covariates and outcomes.
Together, these results reveal that particularities of the performative prediction problem can enable us to recover the causal effect of predictions from observational data.
In particular, we show that, under these conditions, standard supervised learning techniques can be used to ﬁnd these transferable functional relationships by treating predictions as model inputs.
Empirically, we demonstrate that supervised learning succeeds in ﬁnding M Y even in ﬁnite samples.
We conclude with a discussion of limitations and extensions of our work, pointing out potential violations of the modeling assumptions underlying our causal analysis and proposing directions for future work.
1.2 Broader context and related work
The work by Perdomo et al.
[51], initiated the discourse of performativity in the context of supervised learning by pointing out that the deployment of a predictive model can impact the data distribution we train our models on.
Existing scholarship on performative prediction [c.f., 51, 42, 12, 44, 24, 26, 68, 45, 52, 31] has predominantly focused on achieving a particular solution concept with a prediction function that maps X to Y in the presence of unknown performative effects.
We are interested in understanding the underlying causal mechanism of the performative distribution shift.
Our work is motivated by the seemingly natural approach of lifting the supervised-learning problem and incorporating the prediction as an input feature when building a meta machine learning model for explaining Y .
By establishing a connection to causal identiﬁability, our goal is to understand when such a data-driven strategy can help anticipate the down stream effects of predictions
This work focuses on the setting where predictions lead to changes in the relationship between covariates X and label Y , while the marginal distribution P ( X ) over covariates is assumed to be ﬁxed.
This setting where performativity only surfaces in the label describes an interesting subclass of problems falling under the umbrella of performative (aka.
model-induced or decision-dependent) distribution shifts [51, 37, 12].
Our assumptions are complementary to the strategic classiﬁcation framework [8, 20] that focuses on a setting where performative effects concern P ( X ) , while P ( Y | X ) is assumed to remain stable.
Consequently, causal questions in strategic classiﬁcation [e.g., 22, 3, 59] are concerned with identifying stable causal relationships between X and Y .
Since we assume P ( Y | X ) can change (i.e. the true underlying ’concept’ determining outcomes can change), conceptually different questions emerge in our work.
Similar in spirit to strategic classiﬁcation, the work on algorithmic recourse and counterfactual explanations [32, 28, 65] focuses on the causal link between features and predictions, whereas we focus on the down-stream effects of predictions.
There are interesting parallels between our work and related work on the ofﬂine evaluation of online policies [e.g., 35, 63, 36, 58].
In particular, [63] explicitly emphasize the importance of logging propensities of the deployed policy during data collection to be able to mitigate selection bias.
In our work the deployed model can induce a concept shift.
Thus, we ﬁnd that additional information about the predictions of the deployed model needs to be recorded to be able to foresee the impact of a new predictive model on the conditional distribution P
( Y | X ) , beyond enabling propensity weighting [55].
A notable work by [66] investigates how predictions at one time step impact predictions in future time steps.
Complementary to these existing works we show that randomness in the predictive model is not the only way causal effects of predictions can be identiﬁed.
For our theoretical results, we build on classical tools from causal inference [48, 57, 64].
In particular, we distill unique properties of the performative prediction problem to design assumptions for the identiﬁability of the causal effect of predictions.
The causal force of prediction
Predictions can be performative and impact the population of individuals they aim to predict.
Formulized it in the language of causal inference [48]: the deployment of a predictive model represents an intervention on a causal diagram that describes the underlying data generation process of the population.
We will expand on this causal perspective to study an instance of ths performative prediction problem described below.
2.1 Prediction as a partial mediator
Consider a machine learning application relying on a predictive model f that maps features X to a predicted label ˆ Y .
We assume the predictive model f is performative in that the prediction ˆ Y = f
( X ) has a direct causal effect on the outcome variable Y of the individual it concerns.
Thereby the prediction impacts how the outcome variable Y is generated from the features X .
The causal diagram illustrating this setting is visualized in Figure 1.
The features X ∈ X ⊆ R d are drawn i.i.d.
from a ﬁxed underlying continuous distribution over covariates D X with support X .
The outcome Y ∈ Y ⊆
R is a function of X , partially mediated by the prediction ˆ Y ∈ Y .
The prediction ˆ Y is determined by the deployed predictive model f : X → Y .
For a given prediction function f , every individual is assumed to be sampled i.i.d.
from the data generation process described by the causal graph in Figure 1.
We assume the exogenous noise ξ Y is zero mean, and ξ f allows the prediction function to be randomized.
[Equation]
Note that our model is not meant to describe performativity in its full generality (which includes other ways f may affect P ( X, Y ) ).
Rather, it describes an important and practically relevant class of performative feedback problems that are characterized by two properties: 1) performativity surfaces only in the label Y , and 2) performative effects are mediated by the prediction, such that Y ⊥⊥ f | ˆ Y , rather than dependent on the speciﬁcs of the decision rule.
Application examples.
Causal effects of predictions on outcomes have been documented in multiple contexts:
A bank’s prediction about the client (e.g., his or her creditworthiness in applying for a loan) determines the interest rate assigned to them, which in turn changes a client’s ﬁnancial situation [41].
Mathematical models that predict stock prices inform the actions of traders and thus heavily shape ﬁnancial markets and economic realities [38].
Zillow’s housing price predictions directly impact sales prices [39].
Predictions about the severity of an illness play an important role in treatment decisions and hence the very chance of survival of the patient [34].
Another prominent example from psychology is the Pygmalion effect [56].
It refers to the phenomenon that high expectations lead to improved performance, which is widely documented in the context of education [6], sports [61], and organizations [16].
Examples of such performativity abound, and we hope to have convinced the reader that the performative effects in the label are important for algorithmic prediction.
2.2 Implications for performativity-agnostic learning
Begin with considering the classical supervised learning task where ˆ Y is unobserved.
The goal is to learn a model h : X → Y for predicting the label Y from the features X .
To understand the inherent challenge of classical prediction under performativity, we investigate the relationship between X and
Y more closely.
Speciﬁcally, the data generation process (Figure 1) implies that
[Equation]
This expression makes explicit how the relationship between X and Y that we aim to learn depends on the predictive model governing P ( ˆ Y | X ) .
As a consequence, when the deployed predictive model at test time differs from the model at training time, performative effects surface as concept shift [17].
Such distribution shift problems are known to be intractable without structural knowledge about the shift, implying that we can not expect h to generalize to distributions induced by future model deployments.
Let us inspect the resulting extrapolation gap in more detail and put existing positive results on performative prediction into perspective.
Extrapolation loss.
We illustrate the effect of performativity on predictive performance using a simple instantiation of the structural causal model from Figure 1.
Therefore, assume a linear performative effect of strength α > 0 and a base function g 1 : X → Y
[Equation]
Now, assume we collect training data under the deployment of a predictive model f θ and validate our model under the deployment of f φ .
We adopt the notion of a distribution map from Perdomo et al.
[51] and write D XY ( f ) for the joint distribution over ( X, Y ) surfacing from the deployment of a model f .
We assess the quality of our predictive model h : X → Y over a distribution D XY ( f ) induced by f via the loss function (cid:96) :
Y × Y → R and write R f ( h ) :
= E x,y ∼D XY ( f ) (cid:96) ( h ( x ) , y ) for the risk of h on the distribution induced by f .
We use h ∗ f for the risk minimizer h ∗ f := argmin h ∈H R f ( h ) , and H for the hypothesis class we optimize over.
Proposition 1 bounds the extrapolation loss and can be viewed as a concrete instantiation of the more general extrapolation bounds for performative prediction discussed in [37] within the feedback model from Figure 1.
Proposition 1 (Hardness of performativity-agnostic prediction) .
Consider the data generation process in Figure 1 with g given in (5) and f θ , f φ being deterministic functions.
Take a loss function
(cid:96) :
Y × Y → R that is γ -smooth and µ -strongly convex in its second argument.
Let h ∗ f
θ be the risk minimizer over the training distribution and assume the problem is realizable, i.e., h ∗ f θ ∈ H .
Then, we can bound the extrapolation loss of h ∗ f θ on the distribution induced by f φ as
[Equation]
where d 2 D X ( f θ , f φ ) :
=
E x ∼D X ( f θ ( x ) − f φ ( x )) 2 and ∆ R f θ → f φ ( h ) :
=
R f φ ( h ) − R f θ ( h ) .
The extrapolation loss ∆ R f θ → f φ ( h ∗ f θ ) is zero if and only if either the strength of performativity tends to zero ( α → 0 ), or the predictions of the two predictors f θ and f φ are identical over the support of D X .
If this is not the case, an extrapolation gap is inevitable.
This elucidates the fundamental hardness of performative prediction from feature, label pairs ( X, Y ) when performative effects disrupt the causal relationship between X and Y .
The special case where α = 0 aligns with the assumption of classical supervised learning, in which there is no performativity.
This may hold in practice if the predictive model is solely used for descriptive purposes, or if the agent making the prediction does not enjoy any economic power [21].
The second special case where the extrapolation error is small is when d 2 D X
( f θ , f φ ) → 0 .
In which case D XY ( f θ ) and D XY ( f φ ) are equal in distribution and hence exhibit the same risk minimizer.
Such a scenario can happen, for example, if the model f φ is obtained by retraining f θ on observational data and a ﬁxpoint is reached ( f θ
= h ∗ f θ ).
The convergence of policy optimization strategies to such ﬁxpoints (perfromative stablity) has been studied in prior work [e.g., 51, 42, 12] and enabled optimality results even in the presence of performative concept shifts, relying on the target model f φ not being chosen arbitrarily, but based on a pre-speciﬁed update strategy.
Identifying the causal effect of prediction
Having illustrated the hardness of performativity-agnostic learning, we explore under what conditions incorporating the presence of performative predictions into the learning task enables us to anticipate the perfromative effects of ˆ Y on Y .
Towards this goal, we assume that the mediator ˆ Y in Figureis observed—the prediction takes on the role of the treatment in our causal analysis and we can not possibly hope to estimate the treatment effect of a treatment that is unobserved.
3.1 Problem setup
Assume we are given access to data points ( x, ˆ y, y )
generated i.i.d.
from the structural causal model in Figure 1 under the deployment of a prediction function f θ .
From this observational data, we wish to estimate the expected potential outcome of an individual under the deployment of an unseen (but known) predictive model f φ .
We note that given our causal graph, the implication of intervening on the function f can equivalently be explained by an intervention on the prediction ˆ Y .
Thus, we are interested in identifying the causal mechanism:
[Equation]
Unlike P ( Y | X ) , the mecahnism M Y is invariant to the changes in the predictive model governing
P ( ˆ Y | X ) .
Thus, being able to identify M Y will allow us to make inferences about the potential outcome surfacing from planned model updates beyond explaining patterns in historical data.
We can evaluate M Y to infer y for any x at ˆ y =
f φ ( x ) for f φ being the model of interest.
For simplicity of notation, we will write D ( f θ ) to denote the joint distribution over ( X, ˆ Y , Y ) of the observed data collected under the deployment of the predictive model f θ .
We say M Y can be identiﬁed, if it can uniquely be expressed as a function of observed data.
More formally:
Deﬁnition 1 (identiﬁability) .
Given a predictive model f , the causal graph in Figure 1, and a set of assumptions A .
We say M Y is identiﬁable from D ( f ) , if for any function h that complies with assumptions
A
and h ( x, ˆ y ) =
M Y ( x, ˆ y ) for pairs ( x, ˆ y ) ∈ supp( D XY ( f ))
it must also hold that h ( x, ˆ y ) =
M Y ( x, ˆ y ) for all pairs ( x, ˆ y ) ∈
X × Y .
Without causal identiﬁability, there might be models h (cid:48) (cid:54) =
M Y that explain the training distribution equally well but do not transfer to the distribution induced by the deployment of a new model.
Causal identiﬁability is crucial for enabling extrapolation.
It quantiﬁes the limits of what we can infer given access to the training data distribution, ignoring ﬁnite sample considerations.
Identiﬁcation with supervised learning.
Identiﬁability of M Y from samples of D ( f θ ) implies that the historical data collected under the deployment of f θ contains sufﬁcient information to recover the invariant relationship (7).
As a concrete identiﬁcation strategy, consider the following standard variant of supervised learning that takes in samples ( x, ˆ y, y ) and builds a meta-model that predicts Y from X, ˆ Y by solving the following risk minimization problem
[Equation]
where H denotes the hypothesis class.
We consider the squared loss for risk minimization because it pairs well with the exogeneous noise ξ Y in (3) being additive and zero mean.
The strategy (8) is an instance of what we term predicting from predictions .
Lemma 2 provides a sufﬁcient condition for the supervised learning solution h SL to recover the invariant causal quantity M Y .
Lemma 2 (Identiﬁcation strategy) .
Consider the data generation process in Figure 1 and a set of assumptions A .
Given a hypothesis class H such that every h ∈ H complies with A and the problem is realizable, i.e., M Y ∈ H .
Then, if M Y is causally identiﬁable from D ( f θ ) given A , the risk minimizer h SL in (8) will coincide with M Y .
3.2 Challenges for identiﬁability
The main challenge for identiﬁcation of M Y from data is that in general, the prediction rule f θ which produces ˆ Y is a deterministic function of the covariates X .
This means that, for any realization of
X , we only get access to one ˆ Y =
f θ ( X ) in the training distribution, which makes it challenging to disentangle the direct and the indirect effects of X on Y .
To illustrate this challenge, consider the function h ( x, ˆ y ) :
= M Y ( x, f θ ( x )) that ignores the input parameter ˆ y and only relies on x for explaining the outcome.
This function explains y equally well and can not be differentiated from
M Y based on data collected under the deployment of a deterministic prediction rule f θ .
The problem is akin to ﬁtting a linear regression model to two perfectly correlated covariates.
More broadly, this ambiguity is due to what is known as a lack of overlap (or lack of positivity) in the literature of causal inference [47, 23].
In the covariate shift literature, the lack of overlap surfaces when the covariate distribution violates the common support assumption and the propensity scores are not well-deﬁned (see e.g., Pan and Yang [46]).
This problem renders causal identiﬁcation and thus data-driven learning of performative effects from deterministic predictions fundamentally challenging.
Proposition 3 (Nonidentiﬁability from deterministic predictions) .
Consider the structural causal model in Figure 1.
Assume Y non-trivially depends on ˆ Y , and the set Y is not a singleton.
Then, given a deterministic prediction function f , the mechanism M Y is not identiﬁable from D ( f ) .
The identiﬁability issue persists as long as the two variables X , ˆ Y are deterministically bound and there is no incongruence or hidden structure that can be exploited to disentangle the direct effect of X on Y from the indirect effect mediated by ˆ Y .
In the following, we focus on particularities of prediction problems and show how they allow us to identify M Y .
3.3 Identiﬁability from randomization
We start with the most natural setting that provides identiﬁability guarantees: randomness in the prediction function f θ .
Using standard arguments about overlap [47] we can identify M Y ( x, ˆ y ) for any pair x, ˆ y with positive probability in the data distribution D ( f θ ) from which the training data is sampled.
To relate this to our goal of identifying the outcome under the deployment of an unseen model f φ we introduce the following deﬁnition:
Deﬁnition 2 (output overlap) .
Given two predictive models f θ , f φ , the model f φ is said to satisfy output overlap with f θ , if for all x ∈ X and any subset Y (cid:48) ⊆
Y with positive measure, it holds that
[Equation]
In particular, output overlap requires the support of the new model’s predictions f φ ( x ) to be contained in the support of f θ ( x ) for every potential x ∈ X .
The following proposition takes advantage of the fact that the joint distribution over ( X, Y ) is fully determined by the deployed model’s predictions to relate output overlap to identiﬁcation:
Proposition 4.
Given the causal graph in Figure 1, the mechanism M Y ( x, ˆ y ) is identiﬁable from
D ( f θ ) for any pair x, ˆ y with ˆ y =
f φ ( x ) , as long as f φ is a prediction function that satisﬁes output overlap with f θ .
Proposition 4 allows us to pinpoint the models f φ to which we can extrapolate to from data collected under f θ .
Furthermore, it makes explicit that for collecting data to learn about performative effects, it is ideal to deploy a predictor f θ that is randomized so that the prediction output has full support over Y for any x .
Such a model would generate a dataset that guarantees global identiﬁcation of M Y over X × Y and thus robust conclusions about any future deployable model f φ .
One interesting and relevant setting that satisﬁes this property is the differentially private release of predictions through an additive Laplace (or Gaussian) noise mechanism applied to the output of the prediction function [13].
While standard in the literature, a caveat of identiﬁcation from randomization is that there are several reasons a decision-maker may choose not to deploy a randomized prediction function in performative environments, including negative externalities and concerns about user welfare [29], but also business interests to preserve consumer value of the prediction-based service offered.
In the context of our credit scoring example, random predictions would imply that interest rates are randomly assigned to applicants in order to learn how the rates impact their probability of paying back.
We can not presently observe this scenario, given regulatory requirements for lending institutions.
3.4 Identiﬁability through overparameterization
The following two sections consider situations where we can achieve identiﬁcation, without randomization, from data collected under a deterministic f θ .
Our ﬁrst result exploits incongruences in functional complexity arising from machine learning models that are overparameterized [e.g. 30].
By overparameterization, we refer to the fact that the representational complexity of the model is larger than the underlying concept it needs to describe.
Assumption 1 (overparameterization) .
We say a function f is overparameterized with respect to G over X if there is no function g (cid:48) ∈ G and c ∈ R such that f ( x ) =
c · g (cid:48)
( x ) for all x ∈ X .
A challenge for identiﬁcation is that for deterministic f θ the prediction can be reconstructed from X without relying on ˆ Y , and thus h ( x, ˆ y ) =
M Y ( x, f θ ( x )) can not be differentiated from M Y based on observational data.
However, note that this ambiguity relies on there being an admissable h such that h ( · , ˆ y ) for a ﬁxed ˆ y can represent f θ .
If f θ is overparameterized with respect to the hypothesis class H , this ambiguity is resolved.
Let us make this intuition concrete with an example:
Example 3.1.
Assume the structural equation for y in Figure 1 is g ( x, ˆ y )
=
αx + β ˆ y for some unknown α, β .
Consider prediction functions f θ of the following form f θ ( x ) = γx 2 + ξx for some γ, ξ ≥ 0 .
Consider H be the class of linear functions.
Then, any admissable estimate h ∈ H takes the form h ( x, ˆ y )
=
α (cid:48) x
+ β (cid:48)
ˆ y .
For h to be consistent with observations we need α (cid:48)
+ β (cid:48)
ξ = α + βξ and β (cid:48)
γ = βγ .
This system of equations has a unique solution as long as γ > 0 which corresponds to the case where f θ is overparameterized with respect to H .
In contrast, for γ =
the function h ( x, ˆ y )
=
( α + βξ ) x would explain the training data equally well.
The following result generalizes this argument to separable functions.
Proposition 5.
Consider the structural causal model in Figure 1 where f θ is a deterministic function.
Assume that g can be decomposed as g ( X, ˆ Y )
=
g 1 ( X ) + α ˆ Y for some α > 0 and g 1 ∈ G , where the function class G is closed under addition (i.e. g 1 , g 2 ∈ G ⇒ a 1 · g 1 + a 2 · g
2 ∈ G ∀ a 1 , a 2 ∈ R ).
Let H contain functions that are separable in X and ˆ Y , linear in ˆ Y , and ∀ h ∈ H it holds that h ( · , ˆ y ) ∈ G for a ﬁxed ˆ y .
Then, if f θ is overparameterized with respect to G over the support of D X ,
M Y is identiﬁable from D ( f θ ) .
3.5 Identiﬁability from classiﬁcation
A second ubiquitous source of incongruence that we can exploit for identiﬁcation is the discrete nature of predictions in the context of classiﬁcation.
The resulting discontinuity in the relationship between X and ˆ Y enables us to disentangle M Y from the direct effect of X on Y .
This identiﬁcation strategy is akin to the popular regression discontinuity design [33] and relies on the assumption that all other variables in X are continuously related to Y around the discontinuities in ˆ Y .
[Figure]
Proposition 6.
Consider the structural causal model in Figure 1 where f θ is a deterministic function.
Assume that the structural equation for Y is separable g ( X, ˆ Y ) =
g 1 ( X ) + g 2 ( ˆ Y ) , ∀ X, ˆ Y for some differentiable functions g 1 and g 2 .
Further, suppose X is a continuous random variable and ˆ Y is a discrete random variable that takes on at least two distinct values with non-zero probability.
Then,
M Y is identiﬁable from D ( f θ ) .
Similar to Proposition 5, the separability assumption together with incongruence provides a way to disentangle the direct effect from the indirect effect of X on Y .
Separability is necessary in order to achieve global identiﬁcation guarantees without randomness, the identiﬁcation of entangled components without overlap is fundamentally hard.
Thus, under violations of the separability assumptions, we can only expect the separable components of g to be correctly identiﬁed.
Similarly, a regression discontinuity design only enables the identiﬁcation of the causal effect locally around the discontinuity.
Extrapolation away from the decision boundary to models f φ that are substantially different from f θ increasingly relies on separability to hold true.
Empirical evaluation
We investigate empirically how well the supervised learning solution h SL in (8) is able to identify the causal mechanism M Y from observational data in practical settings with ﬁnite data.
Methodology.
We generated semi-synthetic data for our experiments, using a Census income prediction dataset from folktables.org [11].
Using this dataset as a starting point, we simulate a training dataset and test dataset with distribution shift as follows:
First, we choose two different predictors f θ and f φ to predict a target variable of interest (e.g. income) from covariates X (e.g. age, occupation, education, etc.).
If not speciﬁed otherwise, f θ is ﬁt to the original dataset to minimize squared error, while f φ is trained on randomly shufﬂed labels.
Next, we posit a function g for simulating the performative effects.
Then, we generate a training dataset of ( X, ˆ Y , Y ) tuples from the causal model in Figure 1, using the covariates X from the original data, g, and f θ to generate ˆ Y and Y .
Similarly, we generate a test dataset of ( X, ˆ Y , Y ) tuples, using X, g, f φ .
We assess how well supervised methods learn transferable functional relationships by ﬁtting a model h SL to the training dataset and then evaluating the root mean squared error (RMSE) for regression and the accuracy for classiﬁcation on the test dataset.
In our ﬁgures, we visualize the standard error from 10 replicates with different random seeds and we compare it to an in-distribution baseline trained and evaluated on samples of D ( f φ ) .
If not speciﬁed otherwise we use N = 200 , 000 samples.
4.1 Necessity of identiﬁcation guarantees for supervised learning
We start by illustrating why our identiﬁcation guarantees are crucial for supervised learning under performativity.
Therefore, we instantiate the structural equation g in Figure 1 as
[Equation]
10) with g 1 ( X ) =
β (cid:62) X and ξ Y ∼ N (0 , 1) .
The coefﬁcients β are determined by linear regression on the original dataset.
The hyperparameter α quantiﬁes the performativity strength that we vary in our experiments.
The predictions ˆ Y are generated from a linear model f θ that we modify to illustrate the resulting impact on identiﬁability.
We optimize h SL in (8) over H being the class of linear functions.
[Figure]
We start by illustrating a failure mode of supervised learning in a non-identiﬁability setting (Proposition 3).
Therefore, we let f θ be a deterministic linear model ﬁt to the base dataset ( f θ ( X ) ≈
β (cid:62) X ).
This results in M Y not being identiﬁable from D ( f θ ) .
In Figure 2(a) we can see that supervised learning indeed struggles to identify a transferable functional relationship from the training data.
The meta model returns h SL ( X, ˆ Y ) =
(1 + α ) ˆ Y , instead of identifying g , which leads to a high extrapolation error independent of the strength of performativity.
While we only show the error for one f φ in Figure 2(a), the error grows with the distance d 2 D x
( f θ , f φ ) .
In contrast, when the feature ˆ Y is not included, the supervised learning strategy returns h SL ( X ) =
(1 + α ) β (cid:62) X .
The extrapolation loss of this performativity-agnostic model scales with the strength of performativity (Proposition 1) and is thus strictly smaller than the error of the model that predicts from predictions.
Next, we move to the regime of our identiﬁcation results (Proposition 4-6).
Therefore, we modify the way the predictions in the training data are generated.
In Figure 2(b) we use additive Gaussian noise to determine the predictions as ˆ Y =
f θ ( X ) + η with η ∈ N (0 , σ 2 ) .
In Figure 2(c) we augment the input to f θ with second-degree polynomial features to achieve overparameterization.
In Figure 2(d) we round the predictions of f θ to obtain discrete values.
In all three cases, including ˆ Y as a feature is beneﬁcial and allows the model to match in-distribution accuracy baselines, closing the extrapolation gap that is inevitable for performativity-agnostic prediction.
4.2 Strength of incongruence and ﬁnite samples
We next conduct an ablation study and investigate how the degree of overparameterization and the noise level for randomized f θ impacts the extrapolation performance of supervised learning.
Therefore, we consider the setup in (10) with a general function g 1 .
We ﬁx the level of performativity at α = 0 .
5 for this experiment.
We optimize h SL in (8) over H (which we vary).
In Figure 3(a) we investigate the effect of overparameterization of f θ on the extrapolation error of h SL .
We choose fully connected neural networks with a single hidden layer to represent the functions g 1 , f θ and h SL .
For g 1 and H we take a neural network with m = 3 units in the hidden layer.
The model g 1 is ﬁt it to the original dataset.
We vary the number of units in the hidden layer of f θ , denoted m θ .
As expected, the extrapolation error decreases with the complexity of f θ .
As soon as m θ > m φ there is a signiﬁcant beneﬁt to including predictions as features.
In this regime, M Y becomes identiﬁable as Proposition 5 suggests.
In turn, without access to ˆ Y the model suffers an inevitable extrapolation gap due to a concept shift that is independent of the properties of f θ .
In Figure 2(b) we investigate the effect of the magnitude of additive noise added to the predictions.
Here H and g 1 are linear functions.
We have ˆ Y =
f θ ( X ) + βη with η ∈ N (0 , 1) and we vary the noise level β .
We see that even small amounts of noise are sufﬁcient for identiﬁcation and adding ˆ Y as a feature to our meta-machine lenaring model is effective as soon as the noise in f θ is non-zero.
In Figure 2(c) we ﬁx the noise level at α = 0 .
5 and vary the number of samples N .
We ﬁnd that only moderate dataset sizes are necessary for predicting from predictions to approximate M Y in our identiﬁable settings.
Discussion
This paper focused on identifying the causal effect of predictions on outcomes from observational data.
We point out several natural situations where this causal question can be answered, but we also highlight situations where observational data is not sufﬁciently informative to reason about performative effects.
By establishing a connection between causal identiﬁability and the feasibility of anticipating performative effects using data-driven techniques, this paper contributes to a better understanding of the suitability of supervised learning techniques for explaining social effects arising from the deployment of predictive models in economically and socially relevant applications.
We hope the positive results in this work serve as a message for data-collection : only if predictions are observed, they can be incorporated to anticipate the performative effects of future model deployments.
Thus, access to this information is crucial for an analyst hoping to understand the effects of deployed predictive models, an engineer hoping to foresee consequences of model updates, or a researcher studying performative phenomena.
To date, such data is scarcely available in benchmark datasets, hindering the progress towards a better understanding of performative effects, essential for the reliable deployment of algorithmic systems in the social world.
At the same time we have shown that the deterministic nature of prediction poses unique challenges for causal identiﬁability even if ˆ Y is observed.
Thus, the success of observational designs (as shown in our empirical investigations) is closely tied to the corresponding identiﬁability conditions being satisﬁed.
Our results must not be understood as a green-light to justify the use of supervised learning techniques to address performativity in full generality beyond the scope of our theoretical results.
Limitations and Extensions.
The central assumption of our work is the causal model in Figure 1.
While carving out a rich and interesting class of performative prediction problems that allows us to articulate the challenges of covariates and predictions being coupled, it can not account for all mechanisms of performativity.
This in turn gives rise to interesting questions for follow-up studies.
A ﬁrst neglected aspect is performativity through social inﬂuence .
Our causal model, relies on the stable unit treatment value assumption (SUTVA) [23].
There is no possibility for the prediction of one individual to impact the outcome of his or her peers.
Such an individualistic perspective is not unique to our paper but prevalent in existing causal analyses and model-based approaches to performative prediction and strategic classiﬁcation [e.g., 20, 25, 43, 3, 18, 22].
Spillover effects [cf. 60, 64, 1, 40] are yet unexplored in the context of performative prediction.
Nevertheless, they have important implications for how causal effects should be estimated and interpreted.
In the context of our work they imply that an intervention on f can no longer be explaind solely by changing an individual’s prediction.
As a result, approaches for microfounding performative effect based on models learned from simple, unilateral interventions on an individual’s prediction result in different causal estimates than supervised learning based methods for identiﬁcation as studied in this work.
A preliminary study included in Appendix C shows that data-driven techniques can pick up on interference patterns in the data and beneﬁt from structural properties such as network homophily [19], whereas individualistic modeling misses out on the indirect component arising from neighbors inﬂuencing each other.
A second aspect is performativity in non-causal prediction.
Our model posits that prediction is solely based on features X that are causal for the outcome Y .
This is a desirable situation in many practical applications because causal predictions disincentivize gaming of strategic individuals manipulating their features [43, 3] and offer explanations for the outcome that persist across environments [54, 7].
Nevertheless, non-causal variables are often included as input features in practical machine learning prediction tasks.
Establishing a better understanding for the implications of the resulting causal dependencies due to performativity could be an important direction for future work.
Finally, performative effect can also lead to covariate shift and impact the joint distribution
P ( X, Y ) =
P ( Y | X ) P ( X ) over covariates and labels.
We assumed that performative effects only surface in P ( Y | X ) .
For our theoretical results, this implied that overlap in the X variable across environments is trivially satisﬁed, which enabled us to pinpoint the challenges of learning performative effects due to the coupling between X and ˆ Y .
For establishing identiﬁcation in the presence of a causal arrow f θ → X additional steps are required to ensure identiﬁability.
Acknowledgement
The authors would like to thank Moritz Hardt and Lydia Liu for many helpful discussions throughout the development of this project, Tijana Zrnic, Krikamol Muandet, Jacob Steinhardt, Meena Jagadeesan and Juan Perdomo for feedback on the manuscript, and Gary Cheng for helpful discussions on differential privacy.
We are also grateful for a constructive discourse and valuable feedback provided by the reviewers that greatly helped improve the manuscript.
Paper checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] see Appendix F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
[-2 f θ ( x ) for all x ; it implies that α 1 g 1 ( x ) − α (cid:48) 1 g (cid:48) 
-]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[-This equations gives a unique solution that g (cid:48) 
-]
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]