I NVERSE O NLINE L EARNING : U NDERSTANDING N ONS TATIONARY AND R EACTIONARY P OLICIES
A BSTRACT
Human decision making is well known to be imperfect and the ability to analyse such processes individually is crucial when attempting to aid or improve a decisionmaker’s ability to perform a task, e.g. to alert them to potential biases or oversights on their part.
To do so, it is necessary to develop interpretable representations of how agents make decisions and how this process changes over time as the agent learns online in reaction to the accrued experience.
To then understand the decisionmaking processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem.
By interpreting actions within a potential outcomes framework, we introduce a meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect.
We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models.
Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
I NTRODUCTION
Decision modelling is often viewed through the lens of policy learning (Bain & Sammut, 1995;
Abbeel & Ng, 2004), where the aim is to learn some imitation policy that captures the actions (and thus decisions) of an agent in some structured way (Hüyük et al., 2021a).
Unfortunately, this usually implicitly relies on the assumption that the actions taken are already optimal with respect to some objective, and hence do not change over time.
While this might be appropriate to approximate the stationary policy of an autonomous agent, such assumptions often break down when applying them to more ﬂexible agents that might be learning on-the-ﬂy.
In particular, we may want to model the decision making process of a human decision-maker, whose actions we may see executed over time - this is clearly of great use in a range of ﬁelds including economics and medicine (Hunink et al., 2014; Zavadskas & Turskis, 2011).
In the context of machine learning, they would be considered to be undergoing a process of online learning (Hoi et al., 2018); training while interacting with the environment, where it is to be expected that beliefs and strategies are constantly adapted based on accumulating experience (Elio & Pelletier, 1997).
Additionally, such belief updates are often imperfect; for example, people are known to a priori over-weight the potential impact of rare events when they ﬁrst start a task, before proceeding to neglect them when they do not occur in their experience (Hertwig & Erev, 2009).
If our goal is to analyse human decision-making, it is thus important to be able to model non-stationary policies that are potentially imperfect in two senses: both marginally at each time-step, and in the way that they become adapted to new information.
In this work, instead of relying on the assumption that an agent follows a stationary optimal policy, we consider their actions to be consistent within an online leaning framework where they learn from an incoming sequence of examples (Hoi et al., 2018) and update their policies accordingly, as in
Figure 1.
Thus, instead of assuming that actions are globally optimal, we assume that at any exact time the agent thought that the action was better than the alternatives.
As a natural consequence, we frame the problem of decision modelling as that of inverse online learning; that is, we aim to uncover [Figure] the process through which an agent is learning online.
Note that this does not involve coming up with a new way to do online learning itself, rather we recover how it appears a given agent may have learnt.
We do this by modelling the (non-stationary) policy of the agent using a deep state-space model (Krishnan et al., 2017), allowing us to track the memory of the agent while allowing policies to evolve in a non-parametric and non-prescribed fashion.
We construct an interpretable mapping from the accrued memory to policies by framing the decision-making process within a potential-outcome setting (Rubin, 2005); letting actions correspond to interventions (or treatments ), and assuming that agents make decisions by choosing an action which they perceive to have the highest (treatment) effect.
Contributions.
In this work we make a number of contributions: First, we formalise the inverse online learning problem and a connection between decision modelling, online learning and potential outcomes, highlighting how they complement each other to result in an interpretable understanding of policies (Section 2); Second, we introduce a practical method to estimate the nonstationary policy and update method of an agent given demonstrations (Section 4);
And third, we demonstrate how we can uncover useful practical insights into the decision making process of agents with a case study on the acceptance of liver donation offers (Section 5).
Code is made available at https://github.com/XanderJC/inverse-online , along with the group codebase at https://github.com/vanderschaarlab/mlforhealthlabpub .
P ROBLEM F ORMALISATION
Preliminaries.
Assume we observe a data trajectory 1 of the form D
= { ( X t , A t , Y t ) }
Tt =1 .
Here, X t ∈ X ⊂ R d , denotes a context vector of possible confounders; A t ∈ A = { 0 , 1 } , a binary action or intervention; and Y t ∈ Y , a binary or continuous outcome of interest.
The subscript t indicates a time-ordering of observed triplets; any time-step t is generated by (i) arrival of context x , (ii) an intervention a being performed by an agent according to some policy π t ( x ) and (iii) a corresponding outcome y being observed.
[Figure]
Goal.
We are interested in recovering the agent’s non-stationary policy π t ( x ) =
P ( A = 1 | X = x, H t = h t ) , which depends on the observed context x and can change over time due to the observed history H t
= { ( X k , A k , Y k ) } t − 1 k =1 .
We make the key assumption that the agent acts with the intention to choose an intervention a leading to the best (largest) potential outcome Y ( a ) , but that the perception of the optimal intervention may change over time throughout the agent’s learning process about their environment.
This problem formalisation naturally places us at the intersection of treatment effect inference within the Neyman-Rubin potential outcomes (PO) framework (Rubin, 2005) with the classical (inverse) reinforcement learning (RL) paradigm (Abbeel & Ng, 2004).
More speciﬁcally though the RL-subproblem of logged contextual bandits (Swaminathan &
Joachims, 2015; Atan et al., 2018), where the setting is ofﬂine and the contexts arrive independently.
Perception Versus Reality.
As is standard within the PO framework, we assume that any observation has two potential outcomes Y ( a ) , associated with each intervention (treatment-level), which are generated from some true distribution inherent to the environment, i.e. Y (0) , Y (1) ∼ P e ( ·| X = x ) , yet only the potential outcome associated with the chosen treatment is revealed to the agent.
If the agent had full knowledge of the environment, its optimal policy would be to choose an action as arg max a µ ea ( x ) where µ ea ( x ) =
E P e
[ Y ( a ) | X = x ] , or conversely to choose to perform the intervention only when its expected conditional average treatment effect (CATE)
τ e ( x ) = µ e 1 ( x ) − µ e 0 ( x ) is positive.
In general, we assume that such perfect knowledge of the environment is not available to the agent in many scenarios of practical interest.
Instead, we assume that at any time t , the agent acts according to some perceived model ˜ P of its environment, which induces the perceived expected potential outcomes ˜ µ a ( x ; t ) and associated treatment effect ˜ τ ( x ; t ) .
Here, ˜ µ a ( x ; t ) =
E ˜ P
[ Y ( a ) | X = x, H t = h t ] , is a function not only of the observed context but also the history, such that the agent can update its beliefs according to observed outcomes.
Note that we allow the agent’s model of the environment to be misspeciﬁed; in this case ˜ µ a ( x ; t ) would not converge to µ ea ( x ) even for very long trajectories.
We consider this an important feature, as we assume that human decision-makers generally need to make simpliﬁcations (Grifﬁths et al., 2015).
Formally, we make the following assumption about the agent’s decision making process:
Assumption 1 Mutual Observability: The observable space X contains only and all the information available to the agent at the point of assigning an action.
Assumption 2 Perceived Optimality: An agent assigns the action they think will maximise the outcome.
For a deterministic policy π t , the agent believes π t ( x ) = 1 ⇐⇒ ˜ µ 1 ( x ; t ) > ˜ µ 0 ( x ; t ) .
Given a stochastic agent, the policy π t ( x ) = P ( A t = 1 | x ) is a monotonic increasing function of the agent’s belief over the treatment effect ˜ τ
( x ; t ) .
Assumption 3 Continual Adaptation: The agent may continually adjust their strategy based on experience.
The index in the dataset D represents a temporal ordering which π is non-stationary with respect to, i.e. π i ( x ) may not equal π j ( x ) , i (cid:54) =
j ≥ 1 where π i represents the policy at time step i .
Assumptions 1 and 2 are generally unveriﬁable in practice, yet, they are necessary: there would be little opportunity to learn anything meaningful about a non-stationary policy without them.
Mutual observability is crucial, as assuming that different information streams are available to the observer and the agent would make it impossible to accurately describe the decision-making process.
Assumption 2, albeit untestable, appears very reasonable: in the medical setting, for example, a violation would imply malicious intent – i.e. that a clinician purposefully chooses the intervention with a sub-optimal expected outcome.
Nonetheless, this does mean that we do not consider exploratory agents that are routinely willing to sacriﬁce outcomes for knowledge, a constraint we do not consider too restrictive within the medical applications we have in mind.
Assumption 3, on the other hand, is not at all restrictive; on the contrary, it explicitly speciﬁes a more ﬂexible framework than is usual in imitation or inverse reinforcement learning as the set of policies it describes contains stationary policies.
R ELATED W ORK
Our problem setting is related to, but distinct from, a number of problems considered within related literature.
Effectively, we assume that we observe logged data generated by a learning agent that acts according to its evolving belief over effects of its actions, and aim to solve the inverse problem by making inferences over said belief given the data.
Below, we discuss our relationship to work studying both the forward and the inverse problem of decision modelling, with main points summarised in Table 1.
The Forward Problem: Inferring Prescriptive Models for Behaviour.
The agent, whose (nonstationary) policy we aim to understand, actively learns online through repeated interaction with an environment and thus effectively solves an online learning problem (Hoi et al., 2018).
The most prominent example of such an agent would be a contextual bandit (Agarwal et al., 2014), which learns to assign sequentially arriving contexts to (treatment) arms.
Designing agents that update policies online is thus the inverse of the problem we consider here.
Another seemingly related problem within the RL context is learning optimal policies from logged bandit data (Swaminathan & Joachims, 2015); this is different from our setting as it is prescriptive rather than descriptive .
If the goal was to infer the true τ e ( x ) from the (logged) observational data D (instead of the effect perceived by the agent), this would constitute a standard static CATE estimation problem, for which many solutions have been proposed in the recent ML literature (Alaa & van der Schaar, 2018; Shalit et al., 2017; Künzel et al., 2019).
Note that within this literature, the treatmentassignment policy (the so-called propensity score π ( x ) =
P ( A = 1 | X = x ) ) is usually assumed to be stationary and considered a nuisance parameter that is not of primary interest.
Estimating CATE can also be a pretext task to the problem of developing optimal treatment rules (OTR) (Zhang et al.
2012a;b; Zhao et al., 2012), as g ( x )
= 1 ( τ ( x ) > 0) is such an optimal rule (Zhang et al., 2012b).
The Backward Problem: Obtaining a Descriptive Summary of an Agent.
When no outcomes are observed (unlike in the bandit problem) the task of learning a policy becomes imitation learning (IL), itself a large ﬁeld that aims to match the policy of a demonstrator (Bain & Sammut, 1995; Ho & Ermon, 2016).
Note that, compared to the bandit problem, there is a subtle shift from obtaining a policy that is optimal with respect to some true notion of outcome to one that minimises some divergence from the demonstrator.
While IL is often used in the forward problem to ﬁnd an optimal policy – thereby implicitly assuming that the demonstrator is themselves acting optimally –, it can also, if done interpretably, be used in the backward problem to very effectively reason about the goals and preferences of the agent (Hüyük et al., 2021a).
One way to achieve this is through inverse reinforcement learning (IRL), which aims to recover the reward function that is seemingly maximised by the agent (Ziebart et al., 2008;
Fu et al., 2018; Chan & van der Schaar, 2021).
This need not be the true reward function, and can be interpreted as a potentially more compact way to describe a policy, which is also more easily portable given shifts in environment dynamics.
Our problem formalisation is conceptually closely related to the counterfactual inverse reinforcement learning (CIRL) work of Bica et al.
There the authors similarly aim to explain decisions based on the PO framework, speciﬁcally by augmenting a max-margin IRL (Abbeel & Ng, 2004) approach to parameterise the learnt reward as a weighted sum of counterfactual outcomes.
However, CIRL involves a pre-processing counterfactual estimation step that focuses on estimating the true treatment effects, and implicitly assumes that these are identical to the ones perceived by the agent (i.e. it assumes that the agent has perfect knowledge of the environment dynamics generating the potential outcomes), marking a signiﬁcant departure from our (much weaker) assumptions.
Without the focus on treatment effects, our method could be seen as a generalised non-parametric extension of the inverse contextual bandits of Hüyük et al.
(2021b).
In general, the backward problem is hard to evaluate empirically since information about the true beliefs of agents is not normally available in the data, thus relying on simulation to validate (Chan et al., 2021).
[Table] [Table]
( X i ,A
[Table]
π
[Table]
IRL
[Table]
( X i ,A i )
R obs ( x )
[Table] [Table]
( X i ,A i ,Y i )
R obs ( x )
[Table]
( X t ,A t ,Y t ) π obst ( x )
I NFERRING THE O NLINE L EARNING P ROCESS
Preliminaries: The Forward Problem.
In the forward context, the problem we consider amounts to an agent attempting to, at each time-step, take the action they believe will maximise the outcome, without being aware of the true effect of their intervention.
We assume that the agent believes the potential outcomes are a linear combination of the contextual features such that ˜ µ a ( X t ; t )
= (cid:104)
X t , ω ta (cid:105) , with ω ta ∈ Ω a set of weights for the action a at time t , and (cid:104)· , ·(cid:105) denoting the inner product.
Normally we may consider a linear model an oversimpliﬁcation, but [Figure]
𝐴 "
[Figure]
" 𝑌 " 𝝉 " 𝑴 [Figure] 𝐴 ’ 𝑋 ’ 𝑌 ’ 𝝉 ’ 𝑴 ( 𝐴 ( 𝑋 ( 𝑌 ( 𝝉 (
Agent
Environment
[Figure]
ℎ [Figure] ℎ *
[Figure]
when modelling the thought process of an agent it appears much more reasonable - for example, does a doctor consider higher order terms when weighing up potential outcomes of a patient?
It seems unlikely, especially as clinical guidelines are often given as linear cutoffs (Burgers et al., 2003).
The considered forward problem thus proceeds as follows for every given time step t :
A context X t ∼ P X arrives independently over time.
We make no assumptions on the form of P X , in particular noting that there is no need for any level of stationarity.
Given X t and the current set of weights { ω t 1 , ω t 0 } , the agent predicts their perceived potential outcomes ˜ µ 1 ( X t ; t ) and ˜ µ 0 ( X t ; t ) .
Action A t is assigned by some probabilistic optimal treatment rule π : Y 2 (cid:55)→ [0 , 1] such that P ( A t = 1 | X t ) =
π (˜ µ 1 ( X t ; t ) , ˜ µ 0 ( X t ; t )) , before the associated true outcome is observed.
Based on this new information the agent (potentially stochastically) updates their belief over the response surface for the taken action according to some update function f : Ω 2 ×
X ×
A × Y (cid:55)→ (cid:52) (Ω 2 ) such that ω t +11 , ω t + ∼ f ( ω t 1 , ω t 0 , X t , A t , Y t
To maximise outcomes it is crucial to employ an update function f that maximally captures the available information at each step of the process.
This then allows for an appropriate determination of π that aims to maximise the total (potentially future discounted) outcomes.
4. A M ODEL OF I NVERSE O NLINE L EARNING
Here, we focus on the inverse to the problem outlined above, which, to the best of our knowledge, has received little to no attention in related work.
That is, after observing an agent’s actions, can we recover how they arrived at their decisions, and how this changed over time?
In the following, we use vector notation (cid:126)x t : t (cid:48)
= { x i } t (cid:48)
i =
t with (cid:126)x = (cid:126)x 1: T for all realised variables, with histories (cid:126)h t : t (cid:48) =
{ ( x i , a i , y i ) } t (cid:48)
i =
t .
Note that the previously described forward model assumes a factorisation of the full generative distribution of the observed data given by:
[Equation]
We assume that p ( x t ) and p ( y t | a t , x t ) are governed by some true environment dynamics, independent from the perception of the agent and thus not part of our modelling problem.
This seems trivially true, as the perception of the agent will neither affect the contexts that arrive nor the true outcome conditional on the action being taken.
Our model for the evolving beliefs of an agent with respect to their interaction with the environment revolves around a specialised deep state-space model.
Crucially, this takes the form of a latent random variable M t ∈ R d , which captures the memory of the agent at time t , and can evolve in a ﬂexible way based on the observed history.
We consider the memory to be an efﬁcient summary of the history, which can be interpreted as a sufﬁcient statistic for the beliefs of the agent at a given time.
Naturally, this memory is unobserved and must be learnt in an unsupervised manner.
The introduction of this latent variable leads to an extended probabilistic model with a more structured factorisation of the conditional distribution of the actions, and a model given by:
[Equation]
This model consists of three components: a memory summarisation network S θ : H (cid:55)→ (cid:52) ( M ) , which maps the history into a (distribution over) memory; a (perceived) potential outcome predictor network O θ : X × M (cid:55)→ R 2 , that takes the memory and a context and predicts the perceived potential outcomes; and a treatment rule π θ : Y 2 (cid:55)→ [0 , 1] , that given the potential outcomes (summarised by the perceived treatment effect) outputs a distribution over actions.
Throughout, θ denotes the full set of parameters of the generative model and notation is shared across components to reﬂect that they can all be optimised jointly.
Below, we discuss each component in turn.
Memory Aggregation.
We deﬁne the base structure of S θ to be relatively simple and recursive, and assume the memory at a given time step to be distributed as: M
[Equation]
∼ N (cid:0) µ t , Σ t (cid:1) with µ t , Σ t = MLP ([ M t − 1 , X t − 1 , A t − 1 , Y t − 1 ]) , (3) where µ t and Σ t are two output heads of the memory network S given the history h t − 1 .
It is important to note that the memory network does not get the t -th step information (including the context) when predicting the memory at time t - this means the network cannot provide any speciﬁc predictive power for a given context and is forced to model the general response surfaces.
In many models of behaviour the update function f of the agent is modelled as perfect Bayesian inference (Kaelbling et al., 1998; Colombo & Hartmann, 2017).
While this may seems natural on the surface given the logical consistency of Bayesian inference, it seems unlikely in practice that agents (and especially humans) will be capable of it.
Not least because in all but the simplest cases it will become intractable for computational agents and humans themselves are well known to make simplifying approximations when dealing with past histories (Genewein et al., 2015).
By employing a ﬂexible neural architecture, we remove this relatively restrictive assumption on the memory update process.
Our model then learns exactly how the agent appears to use previous information to update their predictions - allowing us to model common cognitive biases such as overly weighting recent or largely negative events (Hertwig & Erev, 2009).
Outcome Estimation.
Having obtained a memory state m t , the second stage involves predicting the perceived potential outcomes for a context x t with the network O θ (˜ τ t | x t , m t ) .
Given the forward model where the potential outcomes are considered a linear combination of the features this then proceeds again in three steps:
[Equation]
[Equation] [Equation] First the memory m t is decoded into a set of weights for each action using some standard feedforward multi-layer perceptron (MLP), before the potential outcomes are predicted by taking the inner product with the context.
This is preferable to, for example, concatenating and passing both x t and m t through a network since, while it may be potentially more expressive, it loses the connection to the forward model as well as the interpretability that is given by linearity.
As it is, while our model of the potential outcomes is linear at each time step , the memory allows this to ﬂexibly change throughout the course of the history.
The perceived treatment effect is then calculated as the difference between potential outcomes.
However, given the linear nature of the predictions, there is no non-degenerate solution for the ω s.
Thus, to ensure identiﬁability, we set ω t 0 = 0 as a baseline.
Treatment Rule.
Under Assumption 2, the agent should be expected to take an action that maximises the outcome.
There are different possible options for the exact treatment rule, the most obvious being π ( a t | ˜ τ t )
=
(˜ τ t > 0) ; assuming that as long as the treatment effect is positive then the intervention will be taken.
However, in order to maintain differentiability, as well as to allow for more modelling ﬂexibility to capture the stochasitcity of agents, we parameterise the policy as a soft version of the indicator function:
[Equation]
Assuming positive α , this is clearly monotonically increasing, satisfying Assumption 2.
These parameters are learnt by the model and thus allow us to model a ﬂexible threshold with β as well as aleatoric uncertainty in the actions of the agent with α .
4. L EARNING WITH AN I NFERENCE N ETWORK
As with all ﬂavours of deep state space models, exact posterior inference is analytically intractable, making optimisation of the marginal likelihood equally so.
As such, we aim to maximise an Evidence
Lower BOund (ELBO) on this likelihood of the data following standardised variational principles (Blei et al., 2017; Hoffman et al., 2013).
This is achieved by positing a parameterised approximate posterior distribution q φ ( (cid:126)m | (cid:126)h ) , and with a simple application of Jensen’s inequality arrive at:
[Equation]
Maximising this bound is not only useful in terms of the marginal likelihood but also equivalently minimises the Kullback-Leibler divergence between the approximate and true posterior D KL [ q [Equation]
[Equation]
|| p
[Equation]
|
[Equation]
)] , giving us more conﬁdence in our inferred posterior.
𝐴 "
[Equation]
$
[Equation]
"#$
[Equation]
#’ 𝑌 "#’
[Equation]
[Equation]
#’ Memory Aggregation 𝐴 " 𝑋 "
[Equation] [Equation]
"#) 𝐴 "
[Equation]
[Equation]
[Equation]
[Equation] "*) 𝒃 "#) [Equation] [Equation]
Backwards LSTM
[Equation]
Factorising the Posterior for Efﬁcient Inference.
In order to accelerate learning and make inference quick, we factorise the approximate posterior in the same way as the true posterior:
[Equation]
Thus to estimate the approximate posterior,we use a backwards LSTM to create a summary b t =
LSTM ( h t : T ) .
This is then concatenated with the previous memory at time step t − 1 before being passed through a fully connected layer to estimate parameters.
Given that the memory can be seen as a sufﬁcient statistic of the past within the model, it is well known that given this factorisation, conditioning on the memory leads to a RaoBlackwellization of the more general network that would be conditioned on all variables (Krishnan et al., 2017; Alaa & van der Schaar, 2019).
This accelerates learning not only by parameter sharing between φ and θ but also by reducing the variance of the gradient estimates during training.
With the speciﬁed inference network q φ , this leads to an optimisation objective:
[Equation]
This can optimised in a straightforward manner using stochastic variational inference and reparameterised Monte Carlo samples from the approximate posterior distribution.
At each step a sample is drawn from the posterior through which gradients can ﬂow and the ELBO is evaluated.
This can be backpropagated through to get gradients with respect to φ and θ which can then be updated as appropriate using an optimiser of the practitioner’s choice.
C ASE S TUDY : L IVER T RANSPLANTATION A CCEPTANCE D ECISIONS
In this section we explore the explainability beneﬁts of our method for evaluating real decision making.
Given space constraints, we relegate validation of the method on synthetic examples to the appendix, and focus here on a real medical example of accepting liver donation offers for transplantation.
This is an important area where much has been done to improve decision support for clinicians (Volk et al., 2015; Berrevoets et al., 2020), but which often focuses on simply suggesting theoretically optimal actions.
Here, by trying to understand better the decision making process, we hope to be able to provide more bespoke support in the future.
Age
Gender
Height
Weight ABO BMI
Creatinine
INR
Total Bilirubin
MELD Score
Serum Sodium
DIAL_TX
MELD Status
Status
Albumin − 0.15 − 0.1 − 0.05 0 0.05 0.
Accept
Age
Gender
Height
Weight ABO BMI
Creatinine
INR
Total Bilirubin
MELD Score
Serum Sodium
DIAL_TX
MELD Status
Status
Albumin − 0.6 − 0.4 − 0.2 0 0.2 0.
Accept
Age
Gender
Height
Weight ABO BMI
Creatinine
INR
Total Bilirubin
MELD Score
Serum Sodium
DIAL_TX
MELD Status
Status
Albumin − 0 .
5 − 0 .
4 − 0 .
3 − 0 .
2 − 0 .
[Figure]
Organ transplantation is an extreme medical procedure, often considered the last resort for critically ill patients (Merion et al., 2005). Despite the signiﬁcant risks associated with major surgery, transplantation can offer the chance of a relatively normal life (Starzl et al., 1982) and as a result demand for donor organs far outstrips supply (Wynn & Alexander, 2011).
Yet, perhaps surprisingly, a large proportion of organ offers are turned down by clinicians (in consultation with the receiving patients) for various reasons including donor quality or perceived poor compatibility with the recipient (Volk et al., 2015).
This decision faced by clinicians can be (under some simpliﬁcation) described by the process: 1) A donor organ becomes available and is offered to a patient; 2)
The clinician considers the state of the organ, the state of the patient, as well as other information such as the previous reject history of the organ (this is the context X ); 3)
The clinician makes the binary decision to accept/reject the organ (the intervention A ); 4) The patient receives/does not receive the organ and a measure of their outcome is recorded - we measure their survival time as the main outcome of interest (the outcome Y ).
Full details of the dataset, preprocessing steps, and experimental setup can be found in the appendix.
Note that in this example we are not trying to capture the policy of any individual doctor, but rather the general policy of a treatment centre over time and how what they ﬁnd important changes - this is due to the form of the data available, but the method could equally be applied to individual clinicians if that information was made available.
Additionally, while survival time will not be recorded immediately, the most powerful signal in this case is usually a death that follows very soon after the decision, which will be effectively captured in the data.
[Figure]
Evolving Relative Importances.
First, we now examine exactly how our method can explain a nonstationary policy over time.
In Figure 5 we plot the relative weights for covariates in terms of the treatment effect for accepting the organ offer as they change at two-year periods.
The Model for EndStage Liver Disease (MELD) score (Kamath & Kim, 2007) was introduced in 2002 to give an overall rating of how sick a patient is.
On introduction, it was the main consideration when making an acceptance decision, as can be seen clearly in Figure 5.
However, over time this became seen as less important as clinicians started to not want to rely on a single score, with some clinicians even turning down offers for patients with high scores in hopes of a better match, with one study showing that% of the patients who died while waiting for an organ had MELD scores ≥ 15 and had previously declined at least one offer (Lai et al., 2012).
This can also be seen reﬂected in Figureas the importance of the MELD score in accepting the offer decreases and the policy stabilises.
Reactionary Policies
The most important aspect of our method is that we can model the reaction of the agent to their experience on the ﬂy.
For example we can see how observing various outcomes change the opinion of the agent moving forward.
This is demonstrated in Figurewhich plots density estimates of the policy shift (measured as the distance the decision boundary moves) after rejecting the offer and measuring a positive/negative outcome.
Here it can be seen that when an offer is rejected then a negative outcome shifts the decision boundary downwards more than a positive outcome, making it less likely to reject an offer in the next time-step.
Matching Clinical Decisions.
While it should be emphasised that our method is (to the best of our knowledge) the only method with the aim or capability to model the evolving decision process of the agent, as mentioned in Section 3 there are a variety of methods that are tangentially related.
First, we consider a few standard approaches to imitation learning with behavioural cloning (BC) (Bain & Sammut, 1995), where we consider both a linear ( BC-Linear ) and deep ( BC-Deep ) functionapproximator, representing both ends of the interpretability spectrum for BC.
Continuing in imitation learning, we also compare alongside reward-regularised classiﬁcation for apprenticeship learning ( RCAL )
(Piot et al., 2014), a method that regularises the reward implied by the learnt Q -values.
We additionally consider an adaptation of counterfactual inverse reinforcement learning ( CIRL )
(Bica et al., 2021) to the bandit setting where we ﬁrst estimate the true CATE of the interventions and thus the true potential outcomes.
We use these as the reward and apply an optimal policy, which in this one-step case is trivially achieved by choosing the action that maximises the potential outcome.
set of demonstrations operator characteristic
[Table]
In Table 2 we compare the predictive power of our method and all benchmarks on the task of imitation learning – i.e. matching the actions of the real demonstrator in some held-out test set.
This is the only task for which we can make meaningful comparison with the existing literature, given our divergent goals.
Despite the fact that at each step our method only considers a linear decision boundary, we are still able to outperform the deep network architectures used in BC and RCAL in terms of
AUC and APS.
This can be put down to the ﬂexible way in which the policy can change over time to allow for adaptation.
Less surprisingly, we also outperform the stationary linear classiﬁer as well as the version of CIRL, the poor performance of which is probably explained by the unrealistic nature of the causal assumptions in this setting.
Further experimental results on action matching in on multiple additional datasets are provided in the appendix.
D ISCUSSION
Limitations.
By employing a deep architecture our method is able to learn a non-parametric and accurate form of both the non-stationary policy and update function, making far fewer assumptions on their true form than is normal in the literature.
This does mean, however, that it works best when training data is abundant, and can otherwise be prone to over-ﬁtting as is common with deep neural networks.
Additionally, the assumption that agents are always taking the action they believe will maximise the outcome does mean our method is out-of-the-box unable to account for strategies that deliberately pick actions that could be sub-optimal but may yield useful information (i.e. exploratory strategies).
An interesting future direction would include exploring this, potentially by augmenting the outcome to include a measure of expected information.
Societal Impact and Ethics.
Being able to better understand decision making processes could be highly useful in combating biases or correcting mistakes made by human decisionmakers.
Nonetheless, it should be noted that any method that aims to analyse and explain observed behaviour has the potential to be misused, for example to unfairly attribute incorrect intentions to someone.
As such, it is important to emphasise that, as is the nature of inverse problems, we are only giving one plausible explanation for the behaviour of an agent and cannot suggest that this is exactly what has gone on in their mind.
Conclusions.
In this paper we have tackled the problem of inverse online learning – that is descriptively modelling the process through which an observed agent adapts their policy over time – by interpreting actions as targeted interventions, or treatments.
We drew together policy learning and treatment effect estimation to present a practical algorithm for solving this problem in an interpretable and meaningful manner, before demonstrating how this could be using in the medical domain to analyse the decisions of clinicians when choosing whether to accept offers for transplantation. A CKNOWLEDGEMENTS
AJC would like to acknowledge and thank Microsoft Research for its support through its
PhD Scholarship Program with the EPSRC.
AC gratefully acknowledges funding from AstraZeneca. This work was additionally supported by the Ofﬁce of Naval Research (ONR) and the NSF (Grant number: 1722516). We would like to thank all of the anonymous reviewers on OpenReview, alongside the many members of the van der Schaar lab for their input, comments and suggestions at various stages that have ultimately improved the manuscript A PPENDIX
Algorithm
Result: Parameters φ of variational distribution and θ of generative model
Input: D , X , A , Y , learning rate η ;
Initialise φ, θ ; while not converged do
Evaluate p θ ( m t | m t − 1 ) ; (cid:46) Calculate forward model next step priors
Evaluate q
[Equation]
m [Equation] t
[Equation]
(cid:126)h ) ; (cid:46) Inference network to get posterior
[Equation]
(cid:46) Straight-through MC sample
[Equation]
(cid:46) Gradient step for φ, θ
[Equation]
endReturn: φ, θ
A E XPERIMENTAL D ETAILS
All experiments were performed on a 2016 MacBook Pro, using a 2.9 GHz
Dual-Core Intel Core i5 with 8GB of LPDDR3 RAM and no GPU acceleration.
Code was written in PyTorch Paszke et al.
and no external code was used in the implementation of either the model nor the benchmarks.
Hyperparameters were selected through grid search over a validation fold of the training data.
All medical data used has undergone a de-identiﬁcation process and consent was obtained by the relevant curators for their data to be publicly released.
A. UNOS D ATA
Our data focuses on data from the United Network for Organ Sharing (UNOS), the US non-proﬁt that manages their national waiting list and maintains a database and record of transplantations that occur.
We focus on liver-transplantations, where the donor is deceased.
The data is made up of patient details from various centres around the United States.
Of the multiple million records of individual transplantations available, once those with missing information on either the donor or recipient are removed we are left with around 500,000 patients spread across 31 centres.
Available variables are listed in Table
In our action matching experiments we split the data into Train/Validation/Test folds by separating the data by centre.
In particular we use centre CTR23901 with 114,314 patients for training;
CTR124 with 21,067 patients for validation; and ﬁnally CTR279 with 14,295 patients for testing.
For the more exploratory experiments who’s focus is on inference over some data and not extrapolated prediction, we simply train on centre
CTR23901, and examine what is inferred about the treatment of those speciﬁc patients
B A DDITIONAL V ALIDATION
While we focused on an important case study in the main paper, we appreciate the need to provide additional evidence that our model can work outside of one speciﬁc case example.
As such, we now report results on some experiments focusing ﬁrst on a toy (but relevant) synthetic example, followed by performance results on multiple further real datasets.
[Table]
meldstat
[Table]
1a
[Table]
hcc
[Table]
1a
B. S YNTHETIC E XPERIMENT
We consider a toy example of an agent that is learning online about the treatment effect of the interventions they are performing.
We let X
=
R 5 , A
= [0 , 1] , and Y = R , with the true outcomes a linear combination of the features given the action (with weights stochastically samples).
From a sampled prior set of weights in the agents mind, the agent then interacts, estimating the potential outcomes and choosing an action stochastically according to equation 7 with α = 1 , β = 0 .
After every context, action, outcome triple ( x, a, y ), the agent adjusts it’s beliefs over the weights of the potential outcome functions using an online gradient descent method:
[Equation]
11) with λ some learning rate.
Clearly then the policy of the agent is nonstationary and changes over time in reaction to the observed outcomes.
Results.
We simulate 10000 trajectories of length 50.
When applying IOL to this simulated data, we can achieve an accuracy of 99%, meaning 99% of the time we can accurately determine which of the potential outcomes the agent thins will be higher.
This is compared to a stationary estimate of the CATE which can only reach 52% given the fact that the policy is detached from the truth of the treatment effect.
Clearly this is an extreme example designed to fail the existing approaches but it does serve to show how stark the contrast is.
B. F URTHER P ERFORMANCE ON R EAL D ATA
Alongside the synthetic example above, we also provide further results on real medical data, demonstrating improved performance across a variety of domains.
These datasets do not contain a recorded outcome, consequently so we do not include CIRL, and the next state is used as a substitute in IOL.
Results are given in Table 2, and a description of the datasets follows:
Intensive Care Unit.
The ICU data covers the treatment of 23,106 of patients in the intensive care unit from Amsterdam UMC Elbers (2019).
These patients are in a more critical state than those on the general wards while suffering similarly from a variety of conditions and consequently are monitored more frequently, with the database containing around 1 billion clinical observations at varying timesteps down to minute by minute recordings.
The data is aggregated into one hour timesteps and models the treatment of the prescription of antibiotics.
Included are 24 series features focusing on vital signs including heart rate, blood pressure, and various chemical blood concentration levels.
Cystic Fibrosis.
The CF data considers patients enrolled in the UK Cystic Fibrosis Registry TaylorRobinson et al.
(2018), coviering around 5,800 patients over the course of multiple years.
Measurements were taken on average every 6 months, and the 79 features cover common comorbidities, test results, and infections that arise.
The intervention corresponds to the prescription of cortico steroids.
Hospital Wards.
The Wards data is based on the care of 6,321 patients at the Ronald Reagan
UCLA Medical Center in California who were treated on the general medicine ﬂoor between 2013-2016 Alaa et al.
These patients were treated for a variety of conditions including pneumonia, sepsis, and fevers were in general stable and deterioration that required ICU care was rare.
Measurements were taken roughly every 4 hours, with average stays lasting 9 days, and include common vital signs such as pulse and blood pressure alongside lab tests and results and 35 temporal features.
The action space is taken as a choice of application of oxygen therapy.
[Table]