A TTENTIVE E XPLAINABILITY FOR P ATIENT T EMPORAL E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning explainable patient temporal embeddings from observational data has mostly ignored the use of RNN architectures that excel in capturing temporal data dependencies but at the expense of explainability.
This paper addresses this problem by introducing and applying an information theoretic approach to estimate and quantify the degree of explainability of such architectures.
Using a communication paradigm, we formalize metrics of explainability by estimating the amount of information that an AI model needs to convey to a human end user to explain and rationalize its outputs.
A key aspect of this work is to model human preferences and prior knowledge at the receiving end and measure a lack of explainability as a deviation from these human preferences.
We apply this paradigm to medical concept representation problems by regularizing loss functions of temporal autoencoders according to the derived explainability metrics to guide the learning process towards models producing explainable outputs.
We illustrate the approach with convincing experimental results for the generation of explainable temporal embeddings for critical care patient data.
I NTRODUCTION
The success of AI models to solve real world problems has been hindered by the inability to fully understand the complex mechanisms behind inferences produced by these models.
This problem is exacerbated in healthcare where clinicians and other stakeholders consuming AI model outputs need to understand the rationales behind the underlying AI inference processes.
Learning explainable representations for patient data has proved to be notoriously difﬁcult and has slowed the adoption of powerful sequence to sequence architectures for this task.
In Choi et al.
(2016a), the authors describe an interesting and effective approach for the embedding of medical concepts.
However, they avoid
RNN like architectures probably able to produce more accurate representations but at the expense of explainability.
Explainability in AI in general has been receiving a great deal of attention within the academic community with numerous methodological advances for the learning of models that are easier to digest by humans.
In Ribeiro et al.
(2016) and Ribeiro et al.
(2018), the authors proposed to locally learn simple linear or tree models to approximate the local performance of complex classiﬁers.
Their approach assumes that explainability is heavily related to model complexity.
In Choi et al.
(2016b), the authors proposed a Recurrent Neural Network architecture able to compute attention coefﬁcients linking the predictions of the model back to the raw features used by the model at different time points.
In this case, explainability is also a local concept that relates to attention mechanisms justifying the predictions made by a model by pointing back to the source data deemed responsible for the predictions.
Recently in Dhurandhar et al.
(2018), the authors proposed to relate explainability not only to the presence of input features responsible for an inference but also to the omission of input features (e.g., the man who does not wear glasses).
In Koh & Liang (2017), the authors proposed to use inﬂuence functions to understand the internals of an AI model by tracing back predictions to training data instances that have inﬂuenced it.
While these papers cleverly shed light on how complex AI models are behaving, they all use different informal notions of explainability–may it be complexity, attentiveness linking predictions to input features or training data inﬂuence on predictions.
They all treat explainability as a qualitative characteristic of AI models and do not rely on well deﬁned measurable aspects of explainability.
In this paper, we treat aspects of explainability as quantitative measurable properties of models and propose approaches to measure the degrees of explainability of learned models in a regression setting, with application to computational embedding.
In general, formalizing explainability is quite hard.
In this work, we only focus on speciﬁc aspects of it that are grounded with the following principles:
• Observability: We restrict our efforts to the generation of explanations as seen by an external observer witnessing the input-output relationship for a given AI system.
This principle is restrictive as it treats any AI agent as a black box.
However, we deem it to be a reasonable assumption that humans commonly make, specially in physical sciences when modeling complex physical phenomena from observational data.
• Communicability: We consider explainability to be a communication problem, with a dialog between an external AI observer and a rational human actor with the external AI observer explaining the behavior of the AI system to the rational human actor at the receiving end of this communication.
The amount of information that needs to be produced by the AI external observer for explainability is consequently inversely proportional to the degree of explainability of the AI system.
• Subjectivity: We view explainability as a subjective concept that depends on the mind set of the receiving rational human actor, in terms of his/her preferences and amount of prior knowledge that he/she may have acquired.
This principle prompts us to model some notion of preferences and/or assumed prior knowledge available to these human actors with prior distributions.
In these cases, AI models are deemed less explainable as they produce predictions based on facts that are harder to explain using generally accepted human prior knowledge.
Based on these principles, we model explainability using information theoretic constructs measuring the amount of information that need to be transmitted by an AI agent to a human end user to explain the rationales behind the predictions.
The main contributions of this paper are three fold: (i) an information theoretic framework to effectively measure aspects of explainability, (ii) the application of the framework to regression problems using attention mechanisms, and (iii) conclusive experimental results on patient temporal embedding problems.
The rest of the paper is organized according to these three contribution areas.
E XPLAINABILITY WITH THE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE 2. N OTATIONS
Throughout the paper, scalars are represented with lower case letters in a non bold typeface, e.g., x .
Sets are represented with upper case letters in a calligraphic typeface, e.g., H .
Vectors are denoted in bold lower case letters e.g., x 2 R k from some positive k 2 N .
Matrices are represented in bold upper case letters, e.g., A 2 R n ⇥ m , where both n, m 2 N .
In general, we use subscripts to index elements in a vector or in a matrix.
For instance, x i is a scalar corresponding to the i th element of vector x while A i,j denotes the i, j entry of matrix A .
The data elements in this paper are always attached to a speciﬁc entity (e.g., a patient in healthcare).
To avoid notation clutter, we often drop these references to entities and describe the methodology at the entity level.
The data for a given entity is represented by a matrix X 2 R n ⇥ m .
A row X i , 1  i  n represents an m dimensional feature vector collected at discrete time i .
Similarly, the output of an AI model processing X is represented by the matrix Y 2 R n ⇥ k corresponding to a discrete time series of k dimensional output vectors.
2. T HE M INIMUM D ESCRIPTION L ENGTH P RINCIPLE
The Minimum Description Length principle is a formalization of the Occam Razor principle guiding the search for good models that explain data using description lengths (Grünwald (2007),Rissanen (1998)).
A simple version of it uses a two-part coding approach to select the "best" model M opt as the one that minimizes the sum of the length in bits for a description of the model (part one of the code) and the length in bit of the effective description of the data using using the model.
Let M denote the set of all computable models in the Turing sense.
The MDL deﬁnes the best model as:
[Equation]
where L is a function that returns description lengths.
The MDL approach imposes preferences for models that balance their complexity with their goodness of ﬁt.
The ﬁrst part of the code is essentially the model complexity while the second part measures the goodness of ﬁt for the model on the data.
An idealized version of the MDL has been deﬁned in Vitányi & Li (2000) where L is replaced with
K , the Kolmogorov complexity.
However, this version is not computable in the Turing sense.
In the following sections, we present practical methods to compute estimate L for explainability purposes.
2. E XPLAINABILITY M ETRICS
Following the observability and communicability assumptions described in Section 1 and the MDL principle, we measure the explainability of an AI model M with description lengths of an external observer model only able to perceive the inputs and outputs of M to model this relationship (See
Figure 1).
In this setting, an explanation M o for an AI model M is essentially such an observer model describing how its inputs X and outputs Y are related in a language that is understandable by human actors.
Assuming the presence of no additional knowledge for M o to characterize this input output relationship, we measure the effectiveness of M o with two part codes as follows:
Deﬁnition 1.
The explainability provided by an observer M o of model M is deﬁned as:
[Equation]
The knowledge agnostic explainability of a model M is deﬁned as:
[Equation]
As stated in Section 1, we also assume that explainability can be subjective and dependent on the mind set of receiving rational human actors, in terms of the amount of prior knowledge that they have acquired.
In our framework, this prior knowledge is represented as a prior model M p that these human actors would use to explain the ( X , Y ) relationship.
For instance, an expert physician may explain the general loss of health of a critical care patient from features representing reductions in heart rate variability in the source data.
For this physician, an AI model producing rationales that are inline with this prior knowledge would certainly by easier to explain than other AI models using variables that are further from her/his mental mind set.
Consequently, we modify the complexity term of the previous deﬁnition to come up with the following measures of explainability in the presence of prior knowledge.
Deﬁnition 2.
With a computable prior knowledge M p , the explainability provided by an observer
M o of model M is deﬁned as:
[Equation]
The knowledge aware explainability of a model M is deﬁned as:
[Equation]
While we have not put any restrictions on the class of models for M o , practical computability considerations prompt us to focus on models that can be algorithmically derived and with structural properties that are understandable by humans (e.g., regression trees, linear regression models).
2. E XPLAINABILITY WITH A TTENTION M ODELS FOR R EGRESSION P ROBLEMS
E ag and E
aw are not computable in the general Turing sense of the word.
However, with restrictions on the class M of models, approximations for these quantities can be deﬁned.
In practice, these restrictions are typically dictated by the problem at hand.
In the rest of this paper, we focus exclusively on regression problems, thus allowing us to tackle the medical concept embedding problem that we are interested in.
Consequently, we restrict M to the set M reg of regression models for all models
M that estimate Y from X , and all observers M o that estimate the ˆ Y =
M ( X ) from X .
With the popularity and effectiveness of attention mechanisms to provide input/output rationales, we further impose attention mechanisms in the architecture of our observers.
This additional step not only allows us to measure the complexity of our observers but also allows us to produce effective explanations.
More speciﬁcally, our observers make use of local attention models estimating at each time i the attention that M is applying to each input feature X ij .
We deﬁne ↵ i ( X i ) =
SoftMax ( ( X i ))
2 R m , where is an arbitrarily complex model producing attention coefﬁcients.
Dense layers are typically used to compute but more complex recurrent neural architectures to compute attention coefﬁcients can also be used as suggested in Choi et al.
(2016b).
M o leverages the ↵ i ’s in the following way:
[Equation]
where P = W
[: , k ]  ↵ i with  denoting the Hadamard product and P =
W [: , k ]  ↵ i being a matrix obtained by stacking the results of the Hadamard product of columns of W with ↵ i .
2.4. E STIMATING T HE G OODNESS OF F IT FOR R EGRESSIVE O BSERVERS
With this restriction to M reg , we model the regression error with a zero mean normal distribution.
This is quite common in regression analysis and also in the application of the MDL to regression problems (Grünwald (2007)).
Hence, for all M o 2 M reg , Y = M o ( X ) + ✏ , where ✏ j ⇠ N (0 ,  2 ) for 1  j  k This restriction allows us to approximate the goodness of ﬁt term L ( M ( X ) | M o , X ) in Equations 2 and 4 for any regressive observer for any AI model without any further assumption on
M .
Since this term essentially measures the amount of information that cannot be explained by the model M o , it tends to have random properties and can be approximated using its Shannon-Fano code (Cover & Thomas (2006),Grünwald (2007)):
[Equation]
using the conditional independence of each ( M ( X ))
i given X .
Clearly, for each i , p ( M ( X ) i | M o ( X ) , X ) follows also a Gaussian distribution N ( M o ( X ) i ,  2 ) .
Hence, the goodness of ﬁt can be rewritten as a scaled mean squared error between M ( X ) and M o ( X ) :
[Equation]
2.4. E STIMATING THE C OMPLEXITY OF L INEAR R EGRESSIVE O BSERVERS
As stated in Grünwald (2007), the basic MDL principle does not deﬁne the expression of the complexity terms in Equations 2 and 4.
These terms are very much dependent on the restricted class of observer models and not on any properties of the data X .
From Equation 6, we are restricted to M o ( X ) =
PX T , a multiplication of the input with the m by k matrix P .
Rows of P can be scaled into attentive distribution that we use to estimate the complexity of M o by measuring the compactness of these rows.
Intuitively, the more compact these row vectors are, the easier it is to explain the corresponding predictions produced by M since the observer would hypothetically need to communicate less bits of information to describe how inputs X relate to outputs in M .
A natural measure of compactness is the Shannon entropy and we estimate the model complexity as follows:
Let Q be a matrix such that Q j = SoftMax ( | P j | ) , then:
[Equation]
where H ( p ) =
P i p
i log 1 p
i
With a conditional on preferences or prior knowledge M p , one may be tempted to measure
L ( M o | M p ) with a conditional entropy, thus extending naturally the approach that we used to estimate L ( M o ) .
However, the computation of such conditional entropy is ill deﬁned as it requires an unknown joint distribution.
Instead, we complement L ( M o ) with the KL divergence between the attentions Q j , 0  j  k and their counterpart computed from prior preferences or knowledge  j , 0  j  k .
Consequently,
[Equation]
In this case, the complexity becomes a simple average cross entropy across all k dimensions of the output.
L EARNING E XPLAINABLE T EMPORAL E MBEDDINGS
Despite the large-scale adoption of Electronic Health Records (EHR) by medical institutions, the secondary re-use of these data sets and its impact in healthcare has been lukewarm at best.
Beyond all the data governance challenges that need to be overcome for proper access and integration, these data sets are quite raw, with very complex data models.
They always need to be reﬁned to become ready for analysis and these data reﬁnement steps are quite expensive, often tailored for speciﬁc applications and not reusable.
Deep representation learning techniques have been proposed by several research groups to tackle these challenges.
However, most of these approaches refrain from using
RNN architectures in attempts to preserve the interpretability of the learned models at the expense of being able to encode efﬁciently the temporal patterns in the data.
Using external regressive operators as observers, we address this issue and present in this section an RNN based framework for the representation learning of temporal embeddings.
3. A N RNN P IPELINE FOR T EMPORAL E MBEDDINGS
Figure 2 illustrates the proposed deep learning architecture.
At a high level, this architecture is an auto-encoder consisting of three components.
First, an RNN encoder M enc is used to transform the input data X into embeddings Y .
We experimented with various RNN cells for the encoder including the popular LSTM and GRU cells.
We eventually settled on the minimalRNN cell presented in Chen (2017) and Chen et al.
because of its predictable dynamics as it is designed to rule out mixing effects across dimensions of the embeddings produced by our autoencoder.
This choice allows us to improve the likelihood of developing a regressive observer model with a reasonable goodness of ﬁt without sacriﬁcing too much on the quality of the embeddings.
The second component is an RNN based decoder M dec that attempts to reconstruct the original data
X from the output of the encoder.
We have equipped this decoder with a temporal attention scheme similar to the ones typically used in NLP sequence 2 sequence architectures for machine translation.
This temporal attention allows the decoder to reconstruct X i using embeddings already computed at previous times i  1 to i  w , w being a hyper-parameter that we keep small (i.e., less than 10).
The third component of this deep learning architecture is the regressive observer M o described in the previous section and used to track the inputs and outputs of the encoder to provide and estimate explainability.
This component is taking X as input and producing an estimate ˆ Y of the embedding Y .
During training, this component may have access to a prior model of user preferences M p that it may use in regularization steps to control deviations from these preferences.
M p allows us to control the contribution of input variables into each dimension of the embeddings produced by the encoder.
To train this deep learning architecture, we use a multi-task approach where we train each of these components jointly.
The joint training of this autoencoder with the observer allows us to softly enforce explainability constraints in this pipeline.
While the role of the observer is essentially to estimate the degree of explainability, we use it here to constrain the generation embeddings towards explainable models according to the metrics that we deﬁned in the previous section.
The encoder and decoder are trained together to minimize the mean square reconstruction error between X and ˆ X , according a loss L rec expressed as:
[Equation]
The regressive observer is trained to provide the best explanations in the MDL sense.
In the absence of prior preferences, its learning is associated with a loss L ag deﬁned as:
[Equation]
[Equation]
Please note that we dropped the term nk 2 log(2 ⇡ 2 ) from the formal expression of e ag ( M o , M enc ) since this term does not have impact on the optimization of the loss function when  is estimated.
In the presence of prior knowledge, the learning of the M o is associated with a loss L aw deﬁned as:
[Equation] [Equation]
Putting all these terms together allows us to deﬁne the overall loss function for this architecture:
[Equation]
in the absence of prior knowledge
[Equation]
in the presence of prior knowledge
3. E XPERIMENTAL R ESULTS
We have performed experiments to demonstrate the effectiveness of our proposed metrics for medical concept embedding context using the MIMIC-III Johnson et al.
(2016) dataset.
For these experiments, we used a cohort of 2395 patients who experienced sepsis, resulting in a total of 109886 data samples.
This data set is sampled on an hourly basis.
Each data sample consists of 37 variables together with a timestamp and a patient ID.
The name of these variables are shown on the Y axis of the plot shown on Figure 3.
The RNN architecture shown in Figure 2 was implemented on tensorﬂow v1.5 (Abadi et al.
Experiments were performed on a cloud environment using a small cluster of K80 GPU units.
Various values of learning rate were tried for the training and the results provided in the paper uses a learning rate of 0.0005 with batch size of 10.
All results were obtained using a 5 fold cross validation approach with partitions computed at the patient level.
A prior model M p was gathered manually to impose soft constraints on how input variables may be grouped together to deﬁne each of the k embedding dimension.
This prior suggests a uniformly distributed focus for each variable in a given group.
This manual grouping led us to deﬁne 16 groups that are shown in Figure 3.
This ﬁgure shows a heatmap of a matrix with rows corresponding to input variables and columns corresponding to embedding dimensions.
Each column shows a suggested distribution of input variables within an embedding dimension.
For instance, the third column suggests a concentration of information that relates to the Glasgow Coma score (a neurological score estimating the consciousness of patients) while the fourth column suggests a concentration of variable measuring various aspects of the cardiovascular system.
Different prior preference models have been tested, providing results that are consistent with the evaluation that we are describing here on a single version of M p .
We focused on evaluating two aspects of the system, namely the qualitatively analysis of M o and the analysis of the impact of explainability regularization on L rec
Figures 4, 5 qualitatively illustrate the explanations provided by M o .
Figure 4 focuses on the knowledge agnostic case and shows heat maps of the spatial attention coefﬁcients for each embedding dimensions for  ag
= 1 and  ag = 10 .
It is clear that increasing  ag adds sparsity on the heat maps and help provide more explainable rationales.
We also looked at the actual input variables that are selected in each of the dimensions.
With  ag = 10 , most embedding dimensions are dominated by a single variable except for the 13th dimension focusing heavily on the blood pressure variables and the ﬁrst dimension focusing on respiration rate and mean arterial pressure.
Figure 5 focuses on the knowledge aware case and shows heat maps of the spatial attention coefﬁcients for each embedding dimensions for  aw = 1 and  aw = 5 .
A quick look at these plots shows that the system is indeed attempting to learn explanations that are in accordance with the prior preferences imposed by regularization.
As  aw increases, M o converges more and more towards M p .
To analyze the impact of explainability regularization on the overall loss, we looked at the R squared associated with the overall reconstruction error and the R squared associated with the ﬁtness of M o , both as a function of regularization parameters.
Table 1 shows R squared results obtained for various values of the
aw .
We can notice that the R squared penalty for the reconstruction is maximized at  aw = 0 .
However, as  aw increases, the R squared remains mostly ﬂat.
In general, we noticed in our experiments that lower values of
aw do not necessarily yield better R squared despite reduction in entropy on the columns of W .
This observation could be a manifestation of the Occam Razor principle that gives preferences for simpler and less complex models that focus on less variables.
However, the penalty is much more severe on the goodness of ﬁt of M o as we vary
aw .
As  aw increases, the complexity of the M o decreases as shown in Figure 5 but this reduction of complexity limits M o ’s ability to effectively estimate M .
The explanations provided by the system become trivial for the complexity of the task at hand.
[Table]
between X and ˆ
[Table]
R squared between Y and ˆ Y
[Table]
C ONCLUDING R EMARKS
We have presented an MDL based approach to estimate the explainability of AI models while taking into account prior preferences.
We have shown how these concepts can be used to regularize the learning of temporal embeddings with regularization terms constraining the production of explainable models, according to metrics that we deﬁned.
Experimental results on a real patient data set from
MIMICIII has demonstrated the applicability of the proposed metrics for the generation of explainable models trading explanation complexity for overall accuracy.
In the future, we plan to run further experiments on larger EHR data sets for other predictive tasks, with prior knowledge extracted from medical guidelines and ontologies.
Furthermore, we plan to extend these concepts to reinforcement learning, in a policy-based setting where we constrain the learning of optimal policies with knowledge aware explainability metrics to produce explainable policies