E XPLAINING BY I MITATING : U NDERSTANDING D ECISIONS BY I NTERPRETABLE P OLICY L EARNING
A BSTRACT
Understanding human behavior from observed data is critical for transparency and accountability in decision-making.
Consider real-world settings such as healthcare, in which modeling a decision-maker’s policy is challenging—with no access to underlying states, no knowledge of environment dynamics, and no allowance for live experimentation.
We desire learning a data-driven representation of decisionmaking behavior that (1) inheres transparency by design , (2) accommodates partial observability , and (3) operates completely ofﬂine .
To satisfy these key criteria, we propose a novel model-based Bayesian method for interpretable policy learning (“I NTERPOLE ”) that jointly estimates an agent’s (possibly biased) belief-update process together with their (possibly suboptimal) belief-action mapping.
Through experiments on both simulated and real-world data for the problem of Alzheimer’s disease diagnosis, we illustrate the potential of our approach as an investigative device for auditing, quantifying, and understanding human decision-making behavior.
I NTRODUCTION
A principal challenge in modeling human behavior is in obtaining a transparent understanding of decision-making.
In medical diagnosis, for instance, there is often signiﬁcant regional and institutional variation in clinical practice [1], much of it the leading cause of rising healthcare costs [2].
The ability to quantify different decision processes is the ﬁrst step towards a more systematic understanding of medical practice.
Purely by observing demonstrated behavior, our principal objective is to answer the question: Under any given state of affairs, what actions are (more/less) likely to be taken, and why?
We address this challenge by setting our sights on three key criteria.
First, we desire a method that is transparent by design .
Speciﬁcally, a transparent description of behavior should locate the factors that contribute to individual decisions, in a language readily understood by domain experts [3, 4].
This will be clearer per our subsequent formalism, but we can already note some contrasts: Classical imitation learning—popularly by reduction to supervised classiﬁcation—does not ﬁt the bill, since black-box hidden states of RNNs are rarely amenable to meaningful interpretation.
Similarly, apprenticeship learning algorithms—popularly through inverse reinforcement learning—do not satisfy either, since the high-level nature of reward mappings is not informative as to individual actions observed in the data.
Rather than focusing purely on replicating actions (imitation learning) or on matching expert performance (apprenticeship learning), our chief pursuit lies in understanding demonstrated behavior.
Second, real-world environments such as healthcare are often partially observable in nature.
This requires modeling the accumulation of information from entire sequences of past observations—an endeavor that is prima facie at odds with the goal of transparency.
For instance, in a fully-observable setting, (model-free) behavioral cloning is arguably ‘transparent’ in providing simple mappings of states to actions; however, coping with partial observability using any form of recurrent function [Table] [Table] approximation immediately lands in black-box territory.
Likewise, while (model-based) methods have been developed for robotic control, their transparency crucially hinges on fully-observable kinematics.
Finally, in realistic settings it is often impossible to experiment online—especially in high-stakes environments with real products and patients.
The vast majority of recent work in (inverse) reinforcement learning has focused on games, simulations, and gym environments where access to live interaction is unrestricted.
By contrast, in healthcare settings the environment dynamics are neither known a priori, nor estimable by repeated exploration.
We want a data-driven representation of behavior that is learnable in a completely ofﬂine fashion, yet does not rely on knowing/modeling any true dynamics.
Contributions Our contributions are three-fold.
First, we propose a model for interpretable policy learning (“I NTERPOLE ”)—where sequential observations are aggregated through a decision agent’s decision dynamics (viz.
subjective belief-update process), and sequential actions are determined by the agent’s decision boundaries (viz.
probabilistic belief-action mapping).
Second, we suggest a
Bayesian learning algorithm for estimating the model, simultaneously satisfying the key criteria of transparency, partial observability, and ofﬂine learning.
Third, through experiments on both simulated and real-world data for Alzheimer’s disease diagnosis, we illustrate the potential of our method as an investigative device for auditing, quantifying, and understanding human decision-making behavior.
R ELATED W ORK
We seek to learn an interpretable parameterization of observed behavior to understand an agent’s actions.
Fundamentally, this contrasts with imitation learning (which seeks to best replicate demonstrated policies) and apprenticeship learning (which seeks to match some notion of performance).
Imitation Learning In fully-observable settings, behavior cloning (BC) readily reduces the imitation problem to one of supervised classiﬁcation [5, 11–13]; i.e. actions are simply regressed on observations.
While this can be extended to account for partial observability by parameterizing policies via recurrent function approximation [14], it immediately gives up on ease of interpretability per the black-box nature of RNN hidden states.
A plethora of model-free techniques have recently been developed, which account for information in the rollout dynamics of the environment during policy learning (see e.g. [15–20])—most famously, generative adversarial imitation learning (GAIL) based on statedistribution matching [6, 21].
However, such methods require repeated online rollouts of intermediate policies during training, and also face the same black-box problem as BC in partially observable settings.
Clearly in model-free imitation, it is difﬁcult to admit both transparency and partial observability.
Speciﬁcally with an eye on explainability, Info-GAIL [22, 23] proposes an orthogonal notion of “interpretability” that hinges on clustering similar demonstrations to explain variations in behavior.
However, as with GAIL it suffers from the need for live interaction for learning.
Finally, several model-based techniques for imitation learning (MB-IL) have been studied in the domain of robotics.
[24] consider kinematic models designed for robot dynamics, while [25] and [7] consider (non-)linear autoregressive exogenous models.
However, such approaches invariably operate in fully-observable settings, and are restricted models hand-crafted for speciﬁc robotic applications under consideration.
Apprenticeship Learning In subtle distinction to imitation learning, methods in apprenticeship learning assume the observed behavior is optimal with respect to some underlying reward function.
Apprenticeship thus proceeds indirectly—often through inverse reinforcement learning (IRL) in order to infer a reward function, which (with appropriate optimization) generates learned behavior that matches the performance of the original—as measured by the rewards (see e.g. [8, 26–29]).
These approaches have been variously extended to cope with partial observability (PO-IRL) [9, 30], to ofﬂine settings through off-policy evaluation [31–33], as well as to learned environment models [10].
However, a shortcoming of such methods is the requirement that the demonstrated policy in fact be optimal with respect to a true reward function that lies within an (often limited) hypothesis class under consideration—or is otherwise black-box in nature.
Further, learning the true environment dynamics [10] corresponds to the requirement that policies be restricted to the class of functions that map from unbiased beliefs (cf. exact inference) into actions.
Notably though, [34] considers both a form of suboptimality caused by time-inconsistent agents as well as biased beliefs.
However, perhaps most importantly, due to the indirect, task-level nature of reward functions, inverse reinforcement learning is essentially opposed to our central goal of transparency—that is, in providing direct, action-level descriptions of behavior.
In Section 5, we provide empirical evidence of this notion of interpretability.
Towards I NTERPOLE
In contrast, we avoid making any assumptions as to either unbiasedness of beliefs or optimality of policies.
After all, the former requires estimating (externally) “true” environment dynamics, and the latter requires specifying (objectively) “true” classes of reward functions— neither of which are necessary per our goal of transparently describing individual actions.
Instead,
I NTERPOLE simply seeks the most plausible explanation in terms of (internal) decision dynamics and (subjective) decision boundaries.
To the best of our knowledge, our work is the ﬁrst to tackle all three key criteria—while making no assumptions on the generative process behind behaviors.
Table 1 contextualizes our work, showing typical incarnations of related approaches and their graphical models.
Before continuing, we note that the separation between the internal dynamics of an agent and the external dynamics of the environment has been considered in several other works, though often for entirely different problem formulations.
Most notably, [35] tackles the same policy learning problem as we do in online, fully-observable environments but for agent’s with internal states that cannot be observed.
They propose agent Markov models (AMMs) to model such environment-agent interactions.
For problems other than policy learning, [36–38] also consider the subproblem of inferring an agent’s internal dynamics; however, none of these works satisfy all three key criteria simultaneously as we do.
I NTERPRETABLE P OLICY L EARNING
We ﬁrst introduce I NTERPOLE ’s model of behavior, formalizing notions of decision dynamics and decision boundaries.
In the next section, we suggest a Bayesian algorithm for model-learning from data.
Problem Setup Consider a partially-observable decision-making environment in discrete time.
At each step t , the agent takes action a t ∈ A and observes outcome z t ∈ Z .
1 We have at our disposal an observed dataset of demonstrations D =
{ ( a i 1 , z i 1 , . . .
, a i τ i , z i τ i ) } n i =1 by an agent, τ i being the length of the i -th trajectory (we shall omit indices i unless required).
Denote by h t .
=( a 1 , z 1 , . .
. , a t − 1 , z t − 1 ) the observed history at the beginning of step t , where h 1 .
= ∅ .
Analogously, let H t .
=
( A × Z ) t −indicate the set of all possible histories at the start of step t ,
where H 1 . = {∅} , and let H .
= ∪ ∞ t =1 H t .
A proper policy π is a mapping π ∈ ∆( A ) H from observed histories to action distributions, where π ( a | h ) is the probability of taking action a given h .
We assume that D is generated by an agent acting according to some behavioral policy π b .
The problem we wish to tackle, then, is precisely how to obtain an interpretable parameterization of π b .
We proceed in two steps: First, we describe a parsimonious belief-update process for accumulating histories—which we term decision dynamics .
Then, we take beliefs to actions via a probabilistic mapping—which gives rise to decision boundaries .
Decision Dynamics We model belief-updates by way of an input-output hidden Markov model (IOHMM)
identiﬁed by the tuple ( S, A, Z, T, O, b 1 ) , with S being the ﬁnite set of underlying states.
T ∈ ∆( S ) S
×
A denotes the transition function such that T ( s t +1 | s t , a t ) gives the probability of transitioning into state s t +1 upon action a t in state s t , and O ∈ ∆( Z ) A × S denotes the observation function such that O ( z t | a t , s t +1 ) gives the probability of observing z t after taking action a t and transitioning into state s t
+1 .
Finally, let beliefs b t ∈ ∆( S ) indicate the probability b t ( s ) that the environment exists in any state s ∈ S at time t , and let b 1 give the initial state distribution.
Note that—unlike in existing uses of the IOHMM formalism—these “probabilities” are for representing the thought process of the human , and may freely diverge from the actual mechanics of the world .
To aggregate observed histories as beliefs, we identify b t ( s ) with P ( s t = s | h t ) —an interpretation that leads to the recursive belief-update process (where in our problem, quantities T, O, b 1 are unknown):
[Equation]
A key distinction bears emphasis: We do not require that this latter set of quantities correspond to (external) environment dynamics—and we do not obligate ourselves to recover any such notion of “true” parameters.
To do so would imply the assumption that the agent in fact performs exactly unbiased inference on a perfectly known model of the environment, which is restrictive.
It is also unnecessary, since our mandate is simply to model the (internal) mechanics of decision-making— which could well be generated from possibly biased beliefs or imperfectly known models of the world.
In other words, our objective (see Equation 3) of simultaneously determining the most likely beliefs (cf. decision dynamics) and policies (cf. decision boundaries) is fundamentally more parsimonious.
[Figure]
Decision Boundaries Given decision dynamics, a policy is then equivalently a map π ∈ ∆( A ) ∆( S ) .
Now, what is an interpretable parameterization?
Consider the three-state example in Figure 1.
We argue that a probabilistic parameterization that directly induces “decision regions” (cf. panel 1b) over the belief simplex is uniquely interpretable.
For instance, strong beliefs that a patient has underlying mild cognitive impairment may map to the region where a speciﬁc follow-up test is promptly prescribed; this parameterization allows clearly locating such regions—as well as their boundaries.
Precisely, we parameterize policies in terms of | A | -many “mean” vectors that correspond to actions:
[Equation]
where η ≥ 0 is the inverse temperature, (cid:107) · (cid:107) the (cid:96) 2 -norm, and µ a ∈ R | S | the mean vector corresponding to action a ∈ A .
Intuitively, mean vectors induce decision boundaries (and decision regions) over the belief space ∆( S ) :
At any time, the action whose corresponding mean is closest to the current belief is most likely to be chosen.
In particular, lines that are equidistant to the means of any pair of actions form decision boundaries between them.
The inverse temperature controls the transitions between such boundaries: A larger η captures more deterministic behavior (i.e. more “abrupt” transitions), whereas a smaller η captures more stochastic behavior (i.e. “smoother” transitions).
Note that the case of η = 0 recovers policies that are uniformly random, and η → ∞ recovers argmax policies.
A second distinction is due: The exponentiated form of Equation 2 should not be confused with typical
Boltzmann [27] or MaxEnt [39] policies common in RL: These are indirect parameterizations via optimal/soft q -values, which themselves require approximate solutions to optimization problems; as we shall see in our experiments, the quality of learned policies suffers as a result.
Further, using q -values would imply the assumption that the agent in fact behaves optimally w.r.t.
an (objectively) “true” class of reward functions—e.g. linear—which is restrictive.
It is also unnecessary, as our mandate is simply to capture their (subjective) tendencies toward different actions—which are generated from possibly suboptimal policies.
In contrast, by directly partitioning the belief simplex into probabilistic “decision regions”, I NTERPOLE ’s mean-vector representation can be immediately explained and understood.
[Equation]
[Figure]
Learning Objective In a nutshell, our objective is to identify the most likely parameterizations T ,
O , b 1 for decision dynamics as well as η , { µ a } a ∈ A for decision boundaries, given the observed data:
[Equation]
Next, we propose a Bayesian algorithm that ﬁnds the maximum a posteriori (MAP) estimate of these quantities.
Figure 2 illustrates the problem setup.
B AYESIAN I NTERPRETABLE P OLICY L EARNING
Denote with θ .
=
( T, O, b 1 , η, { µ a } a ∈ A ) the set of parameters to be determined, and let θ be drawn from some prior P ( θ ) .
In addition, denote with ¯ D = { ( s i 1 , . . .
, s iτ i +1 )
} ni =1 the set of underlying (unobserved) state trajectories, such that D ∪ ¯ D gives the complete (fully-observed) dataset.
Then the complete likelihood—of the unknown parameters θ with respect to D ∪ ¯ D —is given by the following:
[Equation]
where π ( ·|· ) is described by η and { µ a } a ∈ A , and each b t [ · ] is a function of T , O , b 1 , and h t −(Equation 1).
Since we do not have access to ¯ D , we propose an expectation-maximization (EM)-like algorithm for maximizing the posterior P ( θ |D ) =
P ( D| θ ) P ( θ ) / (cid:82) P ( D| θ )
d P ( θ ) over the parameters:
[Table]
[Table] [Table]
do 5:
Compute P ( ¯ D|D , ˆ θ k − 1 ) (cid:46) Appendix A.
Compute ∇ θ Q ( θ ;ˆ θ k − 1 ) at ˆ θ k − 1 (cid:46) Appendix A.
7: ˆ θ k ← ˆ θ k −+ w [ ∇ θ Q ( θ ; ˆ θ k − 1 ) + ∇ θ log P ( θ )]
θ =ˆ θ k −8: while (6) 9:
ˆ θ ← ˆ θ k −10: Output : MAP estim.
ˆ θ .
=
( ˆ T, ˆ O, ˆ b 1 , ˆ η, { ˆ µ a } a ∈ A )
Bayesian Learning Given an initial estimate ˆ θ 0 , we iteratively improve the estimate by performing the following steps at each iteration k :
• “ E-step ”: Compute the expected log-likelihood of the model parameters θ given the previous parameter estimate
ˆ θ k − 1 , as follows:
[Equation]
where we compute the necessary marginalizations of joint distribution P ( ¯ D|D , ˆ θ k − 1 ) by way of a forward-backward procedure (detailed procedure given in Appendix A.1).
• “ M-step ”: Compute a new estimate ˆ θ k that improves the expected log-posterior—that is, such that:
[Equation]
subject to appropriate non-negativity and normalization constraints on parameters, which can be achieved via gradient-based methods (detailed procedure given in Appendix A.2).
We stop when it becomes no longer possible to obtain a new estimate further improving the expected log-posterior —that is, when the “M-step” cannot be performed.
Algorithm 1 summarizes this learning procedure.
Explaining by Imitating Recall our original mandate—to give the most plausible explanation for behavior.
Two questions can be asked about our proposal to “explain by imitating”—to which we now have precise answers:
One concerns explainability, and the other concerns directness of explanations.
First, what constitutes the “most plausible explanation” of behavior?
Now, I NTERPOLE identiﬁes this as the most likely parameterization of that behavior using a state-based model for beliefs and policies—but otherwise with no further assumptions.
In particular, we are only positing that modeling beliefs over states helps provide an interpretable description of how an agent reasons (which we do have ample evidence for 2 )—but we are not assuming that the environment itself takes the form of a state-based model (which is an entirely different claim).
Mathematically, the complete likelihood (Equation 4) highlights the difference between decision dynamics (which help explain the agent’s behavior) and “true” environment dynamics (which we do not care about).
The latter are independent of the agent, and learning them would have involved just the observation likelihoods alone.
In contrast, by jointly estimating T, O, b 1 with η, { µ a } a ∈ A according to both the observation- and action-likelihoods, we are learning the decision dynamics—which in general need not coincide with the environment, but which offer the most plausible explanation of how the agent effectively reasons.
The second question is about directness :
Given the popularity of the IRL paradigm, could we have simply used an (indirect) reward parameterization, instead of our (direct) mean-vector parameterization?
As it turns out, in addition to the “immediate” interpretability of direct, action-level representations, it comes with an extra perk w.r.t.
computability: While it is (mathematically) possible to formulate a similar learning problem swapping out µ for rewards, in practice it is (computationally) intractable to perform in our setting.
The precise difﬁculty lies in differentiating through quantities π ( a t | b t ) —which in turn depend on beliefs and dynamics—in the action-likelihoods (proofs located in Appendix B):
Proposition 1 ( Differentiability with q -Parameterizations ) Consider softmax policies parameterized by q -values from a reward function, such that π ( a | b ) =
e q ∗ ( b,a ) / (cid:80) a (cid:48)
e q ∗ ( b,a (cid:48) ) in lieu of Equation 2.
Then differentiating through log π ( a t | b t ) terms with respect to unknown parameters θ is intractable .
In contrast, I NTERPOLE avoids ever needing to solve any “forward” problem at all (and therefore does not require resorting to costly—and approximate—sampling-based workarounds) for learning:
Proposition 2 ( Differentiability with µ -Parameterizations ) Consider the mean-vector policy parameterization proposed in Equation 2.
Differentiation through the log π ( a t | b t ) terms with respect to the unknown parameters θ is easily and automatically performed using backpropagation through time.
I LLUSTRATIVE E XAMPLES
Three aspects of I NTERPOLE deserve empirical demonstration, and we shall highlight them in turn:
• Interpretability : First, we illustrate the usefulness of our method in providing transparent explanations of behavior.
This is our primary objective here—–of explaining by imitating.
Accuracy : Second, we demonstrate that the faithfulness of learned policies is not given up for transparency.
This shows that accuracy and interpretability are not necessarily opposed.
• Subjectivity : Third, we show I NTERPOLE correctly recovers underlying explanations for behavior—–even if the agent is biased.
This sets us apart from other state-based algorithms.
In order to do so, we show archetypical examples to exercise our framework, using both simulated and real-world experiments in the context of disease diagnosis .
State-based reasoning is prevalent in research and practice: three states in progressive clinical dementia [42, 43], preterminal cancer screening [44, 45], or even—as recently shown—for cystic ﬁbrosis [46] and pulmonary disease [47].
Decision Environments For our real-world setting, we consider the diagnostic patterns for 1, patients during sequences of 6-monthly visits in the Alzheimer’s Disease Neuroimaging Initiative [48] database ( ADNI ).
The state space consists of normal functioning (“NL”), mild cognitive impairment (“MCI”), and dementia.
For the action space, we consider the decision problem of ordering vs. not ordering an MRI test, which—while often informative of Alzheimer’s—is ﬁnancially costly.
MRI outcomes are categorized according to hippocampal volume: { “avg”, “above avg”, “below avg”, “not ordered” } ; separately, the cognitive dementia rating-sum of boxes (“CDR-SB”) result— which is always measured—is categorized as: { “normal”, “questionable impairment”, “mild/severe dementia” } [42].
In total, the observation space therefore consists of the 12 combinations of outcomes.
We also consider a simulated setting to better validate performance.
For this we employ a diagnostic environment ( DIAG ) in the form of an IOHMM with certain (true) parameters T true , O true , b true1 .
Patients fall within diseased ( s + ) and healthy ( s − ) states, and vital-sign measurements available at every step are classiﬁed within positive ( z + ) and negative ( z − ) outcomes.
For the action space, we consider the decision of continuing to monitor a patient ( a = ), or stopping and declaring a ﬁnal diagnosis—and if so, a diseased ( a + ) or healthy ( a − ) declaration.
If we assume agents have perfect knowledge of the true environment, then this setup is similar to the classic “tiger problem” for optimal stopping [49].
Lastly, we also consider a (more realistic) variant of DIAG where the agent’s behavior is instead generated by biased beliefs due to incorrect knowledge T, O, b 1 (cid:54) =
T true , O true , b true1 of the environment ( BIAS ).
Importantly, this generates a testable version of real-life settings where decision-makers’ (subjective) beliefs often fail to coincide with (objective) probabilities in the world.
Benchmark Algorithms Where appropriate, we compare I NTERPOLE against the following benchmarks: imitation by behavioral cloning [5] using RNNs for partial observability ( R-BC ); Bayesian
IRL on POMDPs [9] equipped with a learned environment model ( PO-IRL ); a fully-ofﬂine counterpart [10] of Bayesian IRL ( Off.
PO-IRL ); and an adaptation of model-based imitation learning [7] to partially-observable settings, with a learned IOHMM as the model ( PO-MB-IL ).
Algorithms requiring learned models for interaction are given IOHMMs estimated using conventional methods [50].
Further information on environments and benchmark implementations is found in Appendix C.
An MRI is less likely to be ordered.
An MRI is more likely to be ordered.
An MRI is not ordered.
An MRI is ordered.
Belief updates Final beliefs
Decision boundary
Belief simplex
NL
Dementia
MCI (a) Typical Normal Patient
NL
Dementia
MCI (b) Typical Deteriorating Patient
NL
Dementia
MCI (c) Patient is Diagnosed Late
NL
Dementia
MCI c oun t e rf ac t u a l upd a t e (d) Patient with High-value Tests
Interpretability First, we direct attention to the potential utility of I NTERPOLE as an investigative device for auditing and quantifying individual decisions. Speciﬁcally, modeling the evolution of an agent’s beliefs provides a concrete basis for analyzing the corresponding sequence of actions taken: • Explaining Trajectories .
Figure 3 shows examples of such decision trajectories for four real ADNI patients.
Each vertex of the belief simplex corresponds to one of the three stable diagnoses, and each point in the simplex corresponds to a unique belief (i.e. probability distribution).
The closer the point is to a vertex (i.e. state), the higher the probability assigned to that state.
For instance, if the belief is located exactly in the middle of the simplex (i.e. equidistant from all vertices), then all states are believed to be equally likely.
If the belief is located exactly on a vertex (e.g. directly on top of MCI), then this corresponds to an absolutely certainty of MCI being the underlying state.
Patients (a) and (b) are “typical” patients who ﬁt well to the overall learned policy.
The former is a normally-functioning patient believed to remain around the decision boundary in all visits except the ﬁrst; appropriately, they are ordered an MRI during approximately half of their visits.
The latter is believed to be deteriorating from MCI towards dementia, hence prescribed an MRI in all visits.
• Identifying Belated Diagnoses .
In many diseases, early diagnosis is paramount [51].
I NTERPOLE allows detecting patients who appear to have been diagnosed signiﬁcantly later than they should have.
Patient (c) was ordered an MRI in neither of their ﬁrst two visits—despite the fact that the “typical” policy would have strongly recommended one.
At a third visit, the MRI that was ﬁnally ordered led to near-certainty of cognitive impairment—but this could have been known 12 months earlier!
In fact, among all ADNI patients in the database, 6.5% were subject to this apparent pattern of “belatedness”, where a late MRI is immediately followed by a jump to near-certain deterioration.
• Quantifying Value of Information .
Patient (d) highlights how I NTERPOLE can be used to quantify the value of a test in terms of its information gain.
While the patient was ordered an MRI in all of their visits, it may appear (on the surface) that the third and ﬁnal MRIs were redundant—since they had little apparent affect on beliefs.
However, this is only true for the factual belief update that occurred according to the MRI outcome that was actually observed.
Having access to an estimated model of how beliefs are updated in the form of decision dynamics, we can also compute counterfactual belief updates—that is belief updates that could have occurred if the MRI outcome in question were to be different.
In the particular case of patient (d), the tests were in fact highly informative, since (as it happened) the patient’s CDR-SB scores were suggestive of impairment, and (in the counterfactual)
the doctor’s beliefs could have potentially leapt drastically towards MCI.
On the other hand, among all MRIs ordered for ADNI patients, 19% may indeed have been unnecessary (i.e. triggering apparently insigniﬁcant belief-updates both factually as well as counterfactually).
Evaluating Interpretability through Clinician Surveys
To cement the argument for interpretability we evaluated I NTERPOLE by consulting nine clinicians from four different countries (United States, United Kingdom, the Netherlands, and China) for feedback.
We focused on evaluating two aspects of interpretability regarding our method:
• Decision Dynamics : Whether the proposed representation of (possibly subjective) belief trajectories are preferable to raw action-observation trajectories—that is, whether decision dynamics are a transparent way of modeling how information is aggregated by decision-makers.
• Decision Boundaries : Whether the proposed representation of (possibly suboptimal) decision boudnaries are a more transparent way of describing policies, compared with the representation of reward functions (which is the conventional approach in the policy learning literature).
For the ﬁrst aspect, we presented to the participating clinicians the medical history of an example patient from ADNI represented in three ways using: only the most recent action-observation, the complete action-observation trajectory, as well as the belief trajectory as recovered by I NTERPOLE .
Result :
All nine clinicians preferred the belief trajectories over action-observation trajectories.
For the second aspect, we showed them the policies learned from ADNI by both Off.
PO-IRL
and
I NTERPOLE , which parameterize policies in terms of reward functions and decision boundaries respectively.
Result :
Seven out of nine clinicians preferred the representation in terms of decision boundaries over that offered by reward functions.
Further details can be found in Appendix D
[Table]
[Table]
[Table]
Accuracy Now, a reasonable question is whether such explainability comes at a cost: By learning an interpretable policy, do we sacriﬁce any accuracy?
To be precise, we can ask the following questions:
• Is the belief-update process the same?
For this, the appropriate metric is the discrepancy with respect to the sequence of beliefs—which we take to be (cid:80) t D KL ( b t (cid:107) ˆ b t ) ( Belief Mismatch ).
• Is the belief-action mapping the same?
Our metric is the discrepancy with respect to the policy distribution itself—which we take to be (cid:80) t D KL ( π b ( ·
| b t ) (cid:107) ˆ π ( ·| ˆ b t ))
( Policy Mismatch ).
• Is the effective behavior the same?
Here, the metrics are those measuring the discrepancy with respect to ground-truth actions observed ( Action-Matching ) for ADNI, and differences in stopping ( Stopping Time Error ) for DIAG.
Note that the action-matching and stopping time errors evaluate the quality of learned models in imitating per se, whereas belief mismatch and policy mismatch evaluate their quality in explaining .
The results are revealing, if not necessarily surprising.
To begin, we observe for the ADNI setting in Table 2 that I NTERPOLE performs ﬁrst- or second-best across all three action-matching based metrics; where it comes second, it does so only by a small margin to R-BC (bearing in mind that
R-BC is speciﬁcally optimized for nothing but action-matching).
Similarly for the DIAG setting, we observe in Table 3 that I NTERPOLE performs the best in terms of stopping-time error.
In other words, it appears that little—if any—imitation accuracy is lost by using I NTERPOLE as the model.
Perhaps more interestingly, we also see in Table 3 that the quality of internal explanations is superior— in terms of both belief mismatch and policy mismatch.
In particular, even though the comparators
PO-MB-IL, PO-IRL, and Off.
PO-IRL are able to map decisions through beliefs, they inherit the conventional approach of attempting to estimate true environment dynamics, which is unnecessary— and possibly detrimental—if the goal is simply to ﬁnd the most likely explanation of behavior.
Notably, while the difference in imitation quality among the various benchmarks is not tremendous, with respect to explanation quality the gap is signiﬁcant—where I NTERPOLE has great advantage.
Subjectivity Most signiﬁcantly, we now show that I NTERPOLE correctly recovers the underlying explanations, even if—or perhaps especially if—the agent is driven by subjective reasoning (i.e. with biased beliefs).
This aspect sets I NTERPOLE ﬁrmly apart from the alternative state-based techniques.
[Figure]
show the evolution beliefs explain ground- learned policies in example are made all display similar ef behavior,
I NTERPOLE identiﬁes dynamics (which help explain an agent’s behavior)
[Table]
Consider the BIAS environment:
Here the true environment dynamics are unchanged from DIAG, but no longer coincide with the agent’s decision dynamics.
Speciﬁcally, we let the behavioral policy be generated using erroneous parameters
O ( z − | a = , s + ) <
O true ( z − | a = , s + ) ; that is, the doctor now incorrectly believes the test to have a smaller false-negative rate than it does in reality —thus biasing their beliefs regarding patient states.
Now suppose we wish to recover the decision boundary from the demonstrations—that is, at what conﬁdence threshold does a doctor commit to a healthy diagnosis?
Figure 4 shows that both
I NTERPOLE and PO-MB-IL appear to recover the correct effective behavior : Starting from a neutral prior, doctors tend to stop and issue a “healthy” diagnosis if the ﬁrst two observations return negative signals.
Importantly, I NTERPOLE also correctly locates the conﬁdence target ∼ 90% .
On the other hand, PO-MB-IL—which ﬁrst attempts to estimate the environment’s true parameters—ends up learning a policy on the basis of miscalibrated beliefs, thereby incorrectly explaining the same effective behavior with a lower conﬁdence target ∼ 70% .
Finally, through similarly benchmarked metrics, Table 4 further conﬁrms I NTERPOLE ’s advantage in providing the (correct) explanations.
D ISCUSSION
Three points deserve brief comment in closing.
First, while we gave prominence to several examples of how I NTERPOLE may be used to audit and improve decision-making behavior, the potential applications are not limited to these use cases.
Broadly, the chief proposition is that having quantitative descriptions of belief trajectories provides a concrete language that enables investigating observed actions, including outliers and variations in behavior: On that basis, different statistics and post-hoc analyses can be performed on top of I NTERPOLE ’s explanations (see Appendix C for examples).
Second, a reasonable question is whether or not it is reasonable to assume access to the state space.
For this, allow us to reiterate a subtle distinction.
There may be some “ground-truth” external state space that is arbitrarily complex or even impossible to discover, but—as explained—we are not interested in modeling this.
Then, there is the internal state space that an agent uses to reason about decisions, which is what we are interested in.
In this sense, it is certainly reasonable to assume access to the state space, which is often very clear from medical literature [42–47].
Since our goal is to obtain interpretable representations of decision, it is therefore reasonable to cater precisely to these accepted state spaces that doctors can most readily reason with.
Describing behavior in terms of beliefs over these (already well-understood) states is one of the main contributors to the interpretability of our method.
Finally, it is crucial to keep in mind that I NTERPOLE does not claim to identify the real intentions of an agent: humans are complex, and rationality is—of course—bounded.
What it does do, is to provide an interpretable explanation of how an agent is effectively behaving, which—–as we have seen for diagnosis of ADNI patients—–offers a yardstick by which to assess and compare trajectories and subgroups.
In particular, I NTERPOLE achieves this while adhering to our key criteria for healthcare settings, and without imposing assumptions of unbiasedness or optimality on behavioral policies. A CKNOWLEDGMENTS
This work was supported by the US Ofﬁce of Naval Research (ONR) and Alzheimer’s Research UK (ARUK). We thank the clinicians who participated in our survey, the reviewers for their valuable feedback, and the Alzheimer’s Disease Neuroimaging Initiative for providing the ADNI dataset.
A D ETAILS OF THE A LGORITHM A. F ORWARD -B ACKWARD P ROCEDURE
We compute the necessary marginalizations of the joint distribution P ( ¯ D|D , ˆ θ ) using the forwardbackward algorithm.
Letting x t : t (cid:48)
= { x t , x t +1 , . . .
, x t (cid:48) } for any time-indexed quantity x t , the forward messages are deﬁned as α t ( s ) =
P ( s t = s, a 1: t − 1 , z 1: t − 1 | ˆ θ ) , which can be computed dynamically as
[Equation]
with initial case α 1 ( s ) =
P ( s 1 = s )
=
b 1 ( s ) .
The backward messages are deﬁned as β t ( s ) =
P ( a t : τ , z t : τ | s t =
s, a 1: t − 1 , z 1: t − 1 , ˆ θ ) , which can also be computed dynamically as β t ( s
[Equation]
= P
[Equation]
a t : τ , z t : τ | s t = s, a 1: t − 1 , z 1: t − 1 , ˆ θ ) = (cid:88)
[Equation]
with initial case β τ +1 ( s ) =
P ( ∅| s τ +
= s, a 1: τ , z 1: τ , ˆ θ ) = 1 .
Then
[Equation]
the marginal probability of being in state s at time t given the dataset D and the estimate ˆ θ can be computed as
[Equation]
and similarly, the marginal probability of transitioning from state s to state s (cid:48) at the end of time t given the dataset D and the estimate ˆ θ can be computed as
[Equation]
[-+
-]1 = s (cid:48) , a t , z t | s t
=
s, a 1: t − 1 , z 1: t − 1 , ˆ θ ) × P ( a t +1: τ , z t +1:
τ | s t +
= s (cid:48) , a 1: t , z 1: t , ˆ θ ) =
α t ( s )ˆ π ( a t | b t ) ˆ T ( s (cid:48) | s, a )
ˆ O
( z | a, s (cid:48) )
β t +1 ( s (cid:48) )
∝ α t ( s ) ˆ T ( s (cid:48) | s, a )
ˆ O
( z | a, s (cid:48) )
β t +1 ( s (cid:48) ) .
A. G RADIENT -A SCENT P ROCEDURE
Taking the gradient of the expected log-likelihood Q ( θ ; ˆ θ ) in (5) with respect to the unknown parameters θ =
( T, O, b 1 , η, µ a ∈ A ) ﬁrst requires computing the Jacobian matrix ∇ b t b t (cid:48) for 1 ≤ t < t (cid:48) ≤ τ , where ( ∇ b b (cid:48) ) ij = ∂b (cid:48)
( i ) /∂b
( j ) for i, j ∈ S .
This can be achieved dynamically as ∇ b t b t (cid:48) =
∇ b t +1 b t (cid:48) ∇ b t b t +1 with initial case ∇ b t (cid:48)
b t (cid:48)
=
I , where
[Equation]
t
[Equation]
[Equation]
ij
[Equation]
A.2. P ARTIAL D ERIVATIVES
The derivative of Q ( θ ; ˆ θ ) with respect to T ( s (cid:48) | s, a ) is ∂Q ( θ ; ˆ θ ) ∂T
( s (cid:48) | s, a ) =
[Equation]
, where
[Equation]
= − 2 η ( b t ( j ) − µ a t ( j ))
+ 2 η (cid:88)
a ∈ A e − η (cid:107) b t − µ a (cid:107)(cid:80) a (cid:48) ∈
A e − η (cid:107) b t − µ a (cid:48) (cid:107) 2 ( b t ( j ) − µ a ( j ))
= − 2 η ( b t ( j ) − µ a t ( j ))
+ 2 η (cid:88)
a ∈
A π ( a | b t )( b t ( j ) − µ a ( j )) and ( ∇ T ( s (cid:48) | s,a ) b t (cid:48) +1 )
i 1 = ∂b t (cid:48)
+1 ( i ) ∂T
( s (cid:48) | s, a ) =
∂ ∂T ( s (cid:48) | s, a ) (cid:18) (cid:80) x ∈ S b t (cid:48)
( x ) T ( i | x, a t (cid:48) )
O ( z t (cid:48) | a t (cid:48) , i ) (cid:80) x ∈ S (cid:80) x (cid:48) ∈ S b t (cid:48)
( x ) T ( x (cid:48) | x, a t (cid:48) )
O ( z t (cid:48) | a t (cid:48) , x (cid:48) )
(cid:19)
=
I { a t (cid:48) =
a } (cid:18)
I { i = s (cid:48) } b t (cid:48) ( s ) O ( z t (cid:48) | a, s (cid:48) ) (cid:80) x ∈ S (cid:80) x (cid:48) ∈ S b t (cid:48)
( x ) T ( x (cid:48) | x, a ) O ( z t (cid:48) | a, x (cid:48) )
− b t (cid:48) ( s ) O (
[Equation]
The derivative of Q ( θ ; ˆ θ ) with respect to O ( z | a, s (cid:48) ) is ∂Q ( θ ; ˆ θ )
[Equation]
[Equation]
The derivative of Q ( θ ; ˆ θ ) with respect to b 1 ( s ) is
[Equation]
where ( ∇ b 1 ( s ) b t ) i 1 = ( ∇ b 1 b t ) is .
The derivative of Q ( θ ; ˆ θ ) with respect to η is
[Equation]
=
[Equation]
=
[Equation]
Finally, the derivative of Q ( θ ; ˆ θ ) with respect to µ a ( s ) is ∂Q ( θ ; ˆ θ ) ∂µ a ( s ) =
∂ ∂µ a ( s ) n (cid:88) i =
τ (cid:88) t = log
[Equation]
a
[Equation]
| b t ) = n (cid:88) i = τ (cid:88) t = ∂ ∂µ a
[Equation]
s )
[Equation]
− η
[Equation]
µ
[Equation]
2 − log (cid:88)
a
[Equation]
∈
A e
[Equation]
η
[Equation]
b t
[Equation] µ a [Equation] = n (cid:88) i =
τ (cid:88) t =
(2 η I { a t = a } ( b t ( s ) − µ a ( s )) − 2 ηπ ( a | b t )
( b t ( s ) − µ a ( s ))
=
n (cid:88) i =
τ (cid:88) t = 2 η ( I { a t = a } − π ( a | b t ))
( b t ( s ) − µ a ( s )) .
B P ROOFS OF P ROPOSITIONS B. P ROOF OF P ROPOSITION
First, denote with q ∗ R ∈ R ∆( S ) ×
A the optimal (belief-state) q -value function with respect to the underlying (state-space) reward function R ∈ R S × A , and denote with v ∗ ∈ R ∆( S ) the corresponding optimal value function v ∗ R ( b ) = softmax a
(cid:48) ∈
A q ∗ R ( b, a (cid:48) ) .
Now, ﬁx some component i of parameters θ ; we wish to compute the derivative of log π ( a | b ) with respect to θ i :
[Equation]
where we make explicit here the dependence on R , but note that it is itself a parameter; that is,
R = θ j for some j .
We see that this in turn requires computing the partial derivative ∂q ∗ R ( b, a ) /∂θ
i .
Let γ be some appropriate discount rate, and denote with ρ R ∈ R ∆( S ) ×
A the effective (belief-state) reward ρ R ( b, a ) .
= (cid:80) s ∈ S b ( s ) R ( s, a ) corresponding to R .
Further, let
[Equation]
denote the (belief-state) transition probabilities induced by T and O , where δ is the Dirac delta function such that δ ( b (cid:48) ) integrates to one if and only if b (cid:48) = 0 is included in the integration region.
Then the partial ∂q ∗ R ( b, a ) /∂θ
i is given as follows:
[Equation]
from which we observe that ∂q ∗ R ( b, a ) /∂θ
i is a ﬁxed point of a certain Bellman-like operator.
Speciﬁcally, ﬁx any function f ∈ R ∆( S ) × A ; then ∂q ∗ R ( b, a ) /∂θ
i is the ﬁxed point of the operator
T πR,i : R ∆( S ) ×
A → R ∆( S ) ×
A deﬁned as follows:
[Equation]
a (cid:48)
[Equation]
which takes the form of a “generalized” Bellman operator on q -functions for POMDPs, where for brevity here we have written ρ R,i ( b, a ) to denote the expression ∂∂θ i
ρ R ( b, a ) + γ (cid:82) b (cid:48) ∈ ∆
[Equation]
[Equation]
i P ( b
[Equation]
| b, a ) db (cid:48) .
Mathematically, this means that a recursive procedure can in theory be deﬁned—cf.
“ ∇ q -iteration”, analogous to q -iteration; see e.g. [52]—that may converge on the gradient under appropriate conditions.
Computationally, however, this also means that taking a single gradient is at least as hard as solving POMDPs in general.
Further, note that while typical POMDP solvers operate by taking advantage of the convexity property of ρ R ( b, a ) —see e.g. [53]—here there is no such property to make use of: In general, it is not the case that ρ R,i ( b, a ) is convex.
To see this, consider the following counterexample: Let S .
= { s − , s + } , A .
= { a = } , Z .
= { z − , z + } , T ( s − | s − ,
a = ) =
T ( s + | s + , a = )
=
p = 1 , O ( z − | a = , s − )
=
O ( z + | a = , s + ) = 1 / 4 , b
1 ( s + ) = 1 / 2 , R ( s − , a = ) = 0 , R
( s + , a = )
= 1 / 2 , and γ = 1 / 2 .
For simplicity, we will simply write b instead of b ( s + ) .
Note that:
[Equation]
 
P
[Equation]
z
[Equation]
b
[Equation]
if b
[Equation]
Now, let the elements of θ be ordered such that p is the i -th element, and consider ρ R,i ( b, a ) — evaluated at p = 1 :
[Equation]
Clearly ρ R,i ( b, a = ) cannot be convex since ρ R,i (1 / 2 , a = )
=
and ρ R,i (1 , a = )
=
but ρ R,i (3 / 4 , a = ) > 0 .
B. P ROOF OF P ROPOSITION
In contrast, unlike the indirect q -value parameterization above (which by itself requires approximate solutions to optimization problems), the mean-vector parameterization of I NTERPOLE maps beliefs directly to distributions over actions.
Now, the derivatives of log π ( a | b ) are given as closed-form expressions in Appendices A.1 and A.2.
In particular, note that each b t is computed through a feed-forward structure, and therefore can easily be differentiated with respect to the unknown parameters θ through backpropagation through time:
Each time step leading up to an action corresponds to a “hidden layer” in a neural network, and the initial belief corresponds to the “features” that are fed into the network; the transition and observation functions correspond to the weights between layers, the beliefs at each time step correspond to the activations between layers, the actions themselves correspond to class labels, and the action likelihood corresponds to the loss function (see Appendices A.1 and A.2).
Finally, note that computing all of the forward-backward messages α t and β t in Appendix A.1 has complexity O ( nτS 2 ) , computing all of the Jacobian matricies ∇ b t b t (cid:48) in Appendix A.2 has complexity
O ( nτ 2 S 3 ) , and computing all of the partial derivatives given in Appendix A.2 has complexity at most O ( nτ 2 S 2 AZ ) .
Hence, fully differentiating the expected log-likelihood Q ( θ ; ˆ θ ) with respect to the unknown parameters θ has an overall (polynomial) complexity O ( nτ 2 S 2 max { S, AZ } ) .
C E XPERIMENT P ARTICULARS C. D ETAILS OF D ECISION E NVIRONMENTS
ADNI We have ﬁltered out visits without a CDR-SB measurement, which is almost always taken, and visits that do not occur immediately after the six-month period following the previous visit but instead occur after 12 months or later.
This ﬁltering leaves 1,626 patients with typically three consecutive visits each.
For MRI outcomes, average is considered to be within half a standard deviation of the population mean.
Since there are only two actions in this scenario, we have set η =and relied on the distance between the two means to adjust for the stochasticity of the estimated policy—closer means being somewhat equivalent to a smaller η .
DIAG We set T true ( s − | s − , · ) =
T true ( s + | s + , · ) = 1 , meaning patients do not heal or contract the diseases as the diagnosis progresses, O true ( z − | a = , s + )
=
O true ( z + | a = , s − )
= 0 .
4 , meaning measurements as a test have a false-negative and false-positive rates of 40% , and b true1 ( s + ) = 0 .
Moreover, the behavior policy is given by T
= T true , O =
O true , b 1 = b true1 , η = 10 , µ a
=
( s + )
= 0 .
5 , and µ a − ( s − ) =
µ a + ( s + ) = 1 . 3 .
Intuitively, doctors continue monitoring the patient until they are 90% conﬁdent in declaring a ﬁnal diagnosis.
In this scenario, T and η are assumed to be known.
The behavior dataset is generated as 100 demonstration trajectories.
BIAS We set all parameters exactly the same way we did in DIAG with one important exception: now O ( s − | a = , z + ) = 0 .
2 while it is still the case that O true ( z − | a = , s + )
= 0 .
4 , meaning O (cid:54) =
O true anymore.
In this scenario, b 1 is also assumed to be known (in addition to T and η ) to avoid any invariances between b 1 and O that we have encountered during training.
The behavioral dataset is generated as 1000 demonstration trajectories.
C. D ETAILS OF B ENCHMARK A LGORITHMS
R-BC We train an RNN whose inputs are the observed histories h t and whose outputs are the predicted probabilities ˆ π ( a | h t ) of taking action a
given the observed history h t .
The network consists of an LSTM unit of size 64 and a fully-connected hidden layer of size 64 .
We minimize the crossentropy loss L = − (cid:80) ni =1 (cid:80) τt =1 (cid:80) a ∈
A I { a t = a } log ˆ π ( a | h t ) using Adam optimizer with learning rate 0 .
001 until convergence, that is when the cross-enropy loss does not improve for 100 consecutive iterations.
PO-IRL
The IOHMM parameters T , O , and b 1 are initialized by sampling them uniformly at random.
Then, they are estimated and ﬁxed using conventional IOHMM methods.
The reward parameter R is initialized as ˆ R
( s, a ) =
ε s,a where ε s,a ∼ N (0 , 0 .
Then, it is estimated via
Markov chain Monte Carlo (MCMC) sampling, during which new candidate samples are generated by adding Gaussian noise with standard deviation 0 .
001 to the last sample.
A ﬁnal estimate is formed by averaging every tenth sample among the second set of 500 samples, ignoring the ﬁrst 500 samples.
In order to compute optimal q-values, we have used an off-the-shelf POMDP solver available at https://www.pomdp.org/code/index.html .
Off.
PO-IRL All parameters are initialized exactly the same way as in PO-IRL.
Then, both the
IOHMM parameters T , O , and b 1 , and the reward parameter R are estimated jointly via MCMC sampling.
When generating new candidate samples, with equal probabilities, we have either sampled new T , O , and b 1 from IOHMM posterior (without changing R ) or obtained a new R the same way we did in PO-IRL (without changing T , O , and b 1 ).
A ﬁnal estimate is formed the same way as in
PO-IRL.
PO-MB-IL
The IOHMM parameters T , O , and b 1 are initialized by sampling them uniformly at random.
Then, they are estimated and ﬁxed using conventional IOHMM methods.
Given the IOHMM parameters, we parameterized policies the same way we did in I NTERPOLE , that is as described in (2).
The policy parameters { µ a } a ∈ A are initialized as ˆ µ 0 a ( s ) =
(1 / | S | + ε a,s ) / (cid:80) s (cid:48) ∈ S (1 / | S | + ε a,s (cid:48) )
where ε a,s (cid:48) ∼ N (0 , 0 .
Then, they are estimated according solely to the action likelihoods in (4) using the EM algorithm.
The expected log-posterior is maximized using Adam optimizer with learning rate 0 .
001 until convergence, that is when the expected log-posterior does not improve for 100 consecutive iterations.
I NTERPOLE All parameters are initialized exactly the same way as in PO-MB-IL.
Then, the IOHMM parameters T , O , and b 1 , and the policy parameters { µ a } a ∈ A are estimated jointly according to both the action likelihoods and the observation likelihoods in (4).
The expected log-posterior is again maximized using Adam optimizer with learning rate 0 .
001 until convergence.
C. F URTHER E XAMPLE : P OST - HOC A NALYSES
Policy representations learned by I NTEPOLE provide users with means to derive concrete criteria that describe observed behavior in objective terms.
These criteria, in turn, enable the quantitative analyses of the behavior using conventional statistical methods.
For ADNI, we have considered two such criteria: belatedness of individual diagnoses and informativeness of individual tests.
Both of these criteria are relevant to the discussion of early diagnosis, which is paramount for Alzheimer’s disease [51] as we have already mentioned during the illustrative examples.
Formally, we consider the ﬁnal diagnoses of a patient to be belated if (i) the patient was not ordered an MRI in one of their visits despite the fact that an MRI being ordered was the most likely outcome according to the policy estimated by I NTERPOLE and (ii) the patient was ordered an MRI in a later visit that led to a near-certain diagnosis with at least 90% conﬁdence according to the underlying beliefs estimated by I NTERPOLE .
We consider An MRI to be uninformative if it neither (factually) caused nor could have (counterfactually) caused a signiﬁcant change in the underlying belief-state of the patient, where an insigniﬁcant change is half a standard deviation less than the mean factual change in beliefs estimated by I NTERPOLE .
Having deﬁned belatedness and informativeness, one can investigate the frequency of belated diagnoses and uninformative MRIs in different cohorts of patients to see how practice varies between one cohort to another.
In Table 5, we do so for six cohorts: all of the patients, patients who are over 75 years old, patients with apoE4 risk factor for dementia, patients with signs of MCI or dementia since their very ﬁrst visit, female patients, and male patients.
Note that increasing age, apoE4 allele, and female gender are known to be associated with increased risk of Alzheimer’s disease [54–57].
For instance, we see that uninformative MRIs are much more prevalent among patients with signs of
MCI or dementia since their ﬁrst visit.
This could potentially be because these patients are monitored much more closely than usual given their condition.
[Table]
Alternatively, one can divide patients into cohorts based on whether they have a belated diagnoses or an uninformative MRI to see which features these criteria correlate with more.
We do so in
Table 6.
For instance, we see that a considerable percentage of belated diagnoses are seen among male patients.
[Table]
C. F URTHER E XAMPLE : D ECISION T REES
Clinical practice guidelines are often given in the form of decision trees, which usually have vague elements that require the judgement of the practitioner [58, 59].
For example, the guideline could ask the practitioner to quantify risks, side effects, or improvements in subjective terms such as being signiﬁcant, serious, or potential.
Using direct policy learning, how vague elements like these are commonly resolved in practice can be learned in objective terms.
Formulating policies in terms of IOHMMs and decision boundaries is expressive enough to model decision trees.
An IOHMM with deterministic observations, that is O
( z | a, s (cid:48) )
= 1 for some z ∈ Z and for all a ∈ A, s ∈ S , essentially describes a ﬁnite-state machine, inputs of which are equivalent to the observations.
Similarly, a deterministic decision tree can be deﬁned as a ﬁnite-state machine with no looping sequence of transitions.
The case where the observations are probabilistic rather than deterministic correspond to the case where the decision tree is traversed in a probabilistic way so that each path down the tree has a probability associated with it at each step of the traversal.
Patient arrives.
Test for the disease.
Is there a signiﬁcant risk of disease?
The patient is healthy.
Test for the sub- type of the disease.
The patient has Disease-A.
The patient has Disease-B.
No
Yes
[Figure]
A Decision Tree
INI
HLT
DIS
DSA
DSB b 1 ( INI ) =
1 π ( TST - DIS | b 1 ) =
T ( H LT | I N
I , T S T - D
I
S )
× O ( ·| T S T - D I S , H
LT )
T ( D I S
| I N
I , T S T - D
I
S )
× O ( ·| T S
T - D I S , D
I S )
b 2 ( DIS ) = ?
π ( TST - TYP | b 2 )
= ?
T ( D S A | D
I S , T S T - T
Y P )
× O ( ·| T S T - T Y P , D S
A )
T ( D S B | D
I S , T S T - T
Y P )
× O ( ·| T S
T - T Y P , D
S B ) (
[Figure]
Using direct policy learning, we can infer the risk of disease, b 2 ( DIS ) , and the probability of testing for the sub-type based on the risk, π ( TST - TYP | b 2 ) , which are left vague in (a).
As a concrete example of modeling decision trees in terms of IOHMMs, consider the scenario of diagnosing a disease with two sub-types: Disease-A and Disease-B. Figure 5a depicts the policy of the doctors in the form of a decision tree.
Each newly-arriving patient is ﬁrst tested for the disease in a general sense without any distinction between the two sub-types it has.
The patient is then tested for a speciﬁc sub-type of the disease only if the doctors deem there is a signiﬁcant risk that the patient is diseased.
Note that which exact level of conﬁdence constitutes as a signiﬁcant risk is left vague in the decision tree.
By modeling this scenario using our framework, we can learn: (i) how the risk is determined based on initial test results and (ii) what amount of risk is considered signiﬁcant enough to require a subsequent test for the sub-type.
Let S =
{ INI , HLT , DIS , DSA , DSB } , where INI denotes that the patient has newly arrived, HLT denotes that the patient is healthy, DIS denotes that the patient is diseased, DSA denotes that the patient has Disease-A, and DSB denotes that the patient has Disease-B. Figure 5b depicts the state space S with all possible transitions.
Note that the initial belief b 1 is such that b 1 ( INI )
= 1 .
Let
A = { TST - DIS , TST - TYP , STP - HLT , STP - DSA , STP - DSB } , where TST - DIS denotes testing for the disease, TST - TYP denotes testing for the sub-type of the disease, and the remaining actions denote stopping and diagnosing the patient with one of the terminal states, namely states HLT , DSA , and
DSB .
After taking action a 1 = TST - DIS and observing some initial test result z 1 ∈ Z , the risk of disease, which is the probability that the patient is diseased, can be calculated with a simple belief update:
[Equation]
Moreover, we can say that the doctors are more likely to test for the sub-type of the disease as opposed to stopping and diagnosing the patient as healthy, that is π
[Equation]
| b 2 ) > π b ( STP - HLT | b 2 ) ,
when b 2 ( DIS ) > µ TST - TYP ( DIS ) + µ STP - HLT ( DIS ) assuming µ TST - TYP ( DIS ) > µ STP - HLT ( DIS ) .
Note that there are only two possible actions at the second time step: actions TST - TYP and STP - HLT .
D D ETAILS OF THE C LINICIAN S URVEYS
Each participant was provided a short presentation explaining (1) the ADNI dataset and the decisionmaking problem we consider, (2) what rewards and reward functions are, (3) what beliefs and belief simplices are, and (4) how policies can be represented in terms of reward functions as well as decision boundaries.
Then, they were asked two multiple-choice questions, one that is strictly about representing histories, and one that is strictly about representing policies.
Importantly, the survey was conducted blindly —i.e. they were given no context whatsoever as pertains this paper and our proposed method.
The question slides can be found in Figures 6 and 7.
Essentially, each question ﬁrst states a hypothesis and shows two/three representations relevant to the hypothesis stated.
Then, the participant is asked which of the representations shown most readily expresses the hypothesis.
Here are the full responses that we have received, which includes some additional feedback:
• ClinicianQuestion 1 : C > B > A Question 2 : B
Additional Feedback : The triangle was initially more confusing than not, but the ﬁrst example (100% uncertainty) was helpful.
It isn’t clear how the dots in the triangle are computed.
Are these probabilities based on statistics?
Diagram is always better than no diagram.
• ClinicianQuestion 1 : C > B > A Question 2 : B Additional Feedback : I always prefer pictures to tables, they are much easier to understand.
• ClinicianQuestion 1 : C > B > A Question 2 : B Additional Feedback : Of course the triangle is more concise and easier to look at.
But how is the decision boundary obtained?
Does the decision boundary always have to be parallel to one of the sides of the triangle?
• ClinicianQuestion 1 : C > B > A Question 2 : B
Additional Feedback :
[Regarding Question 1,] representation A and B do not show any interpretation of the diagnostic test results, whereas representation C does.
I think doctors are most familiar with representation B, as it more closely resembles the EHR.
Although representation C is visually pleasing, I’m not sure how the scale of the sides of the triangle should be interpreted.
[Regarding
Question 2,] again I like the triangle, but it’s hard to interpret what the scale of the sides of the triangle mean.
I think option A is again what doctors are more familiar with.
• ClinicianQuestion 1 : C > B > A Question 2 : A • ClinicianQuestion 1 : C > B >
A Question 2 : A • ClinicianQuestion 1 : C Question 2 : B
Additional Feedback : I thought I’d share with you my thoughts on the medical aspects in your scenario ﬁrst
(although I realise you didn’t ask me for them).
The Cochrane review concludes that MRI provides low sensitivity and speciﬁcity and does not qualify it as an add on test for the early diagnosis due to dementia (Lombardi G et.
al.
Cochrane database 2020).
The reason for MRI imaging is (according to the international guidelines) to exclude non-degenerative or surgical causes of cognitive impairment.
In your example the condition became apparent when the CDR-SB score at Month 24 hit 3.0 (supported by the sequence of measurements over time showing worsening CDR-SB score).
I imagine the MRI was triggered by slight worsening in the
CDR-SB score (to exclude an alternative diagnosis).
To answer your speciﬁc questions: Q1.
The representation C describes your (false) hypothesis that it was the MRI that made the diagnosis of MCI more likely/apparent the best—I really like the triangles.
Q2.
I really like the decision boundary.
• ClinicianQuestion 1 : C > B > A Question 2 : B • ClinicianQuestion 1 : C Question 2 : B
Additional Feedback : Q1.
Representation C gives the clearest illustration of the diagnostic change following MRI.
However, the representation of beliefs on a continuous spectrum around discrete cognitive states could be potentially confusing given that cognitive function is itself a continuum (and ‘MCI’, ‘Dementia’ and ‘NL’ are stations on a spectrum rather than discrete states).
Also, while representation C is the clearest illustration, it is the representation that conveys the least actual data and it isn’t clear from the visualisation exactly what each shift in 2D space represents.
Also, the triangulation in ‘C’ draws a direct connection between NL and Dementia, implying that this is a potential alternative route for disease progression, although this is more intuitively considered as a linear progression from NL to MCI to Dementia.
Q2.
For me, the decision boundary representation best expresses the concept of the likelihood of ordering and MRI with the same caveats described above.
Option B does best convey the likelihood of ordering an MRI, but doesn’t convey the information value provided by that investigation.
However, my understanding is that this is not what you are aiming to convey here.