Paper:
iclr2019_conference
Learning explainable patient temporal embeddings from observational data has mostly ignored the use of RNN architectures that excel in capturing temporal data dependencies but at the expense of explainability. This paper addresses this problem by introducing and applying an information theoretic approach to estimate and quantify the degree of explainability of such architectures. Using a communication paradigm, we formalize metrics of explainability by estimating the amount of information that an AI model needs to convey to a human end user to explain and rationalize its outputs. A key aspect of this work is to model human preferences and prior knowledge at the receiving end and measure a lack of explainability as a deviation from these human preferences. We apply this paradigm to medical patient representation problems by regularizing loss functions of temporal autoencoders according to the derived explainability metrics to guide the learning process towards models producing explainable outputs. We illustrate the approach with convincing experimental results for the generation of explainable temporal embeddings for critical care patient data.

Review:
Summary:
The authors propose a framework for training an external observer that tries explain the behavior of a prediction function using the minimal description principle. They extend this idea by considering how a human with domain knowledge might have different expectations for the observer. The authors test this framework on a multi-variate time series medical data (MIMIC-III) to show that, under their formulation, the external observer can learn interpretable embeddings.
Pros:
- Interesting approach: Trying to develop an external observer based on information theoretic perspective. 
- Considering the domain knowledge of the human subject can potentially be an important element when we want to use interpretable models in practice.
Issues:
(1) In 2.4: So between M and M^O, which one is a member of M_reg? 
(2) On a related note to issue (1): In 2.4.1, "Clearly, for each i, (M(X))_i | M^O(X) follows also a Gaussian distribution: First of all, I'm not sure if that expression is supposed to be  p(M(X)_i | M^O(X)) or if that was intended. But either way, I don't understand why that would follow a normal distribution. Can you clarify this along with issue (1)?
(3) In 2.4.2: The rationale behind using attention & compactness to estimate the complexity of M^O is weak. Can you elaborate this in the future version?
(4) What do each figure in Figure 4 represent?
(5) More of a philosophical question: the authors train M and M^O together, but it seems more appropriate to train an external observer separately. If we are going to propose a framework to train an agent that tries to explain a black box function, then training the black-box function together with the observer can be seen as cheating. It can potentially make the job of the observer easier by training the black box function to be easily explainable. It would have been okay if this was discussed in the paper, but I can't find such discussion.
(6) The experiments can be made much stronger by applying this approach to a specific prediction task such as mortality prediction. The current auto-encoding task doesn't seem very interesting to apply interpretation.
(7) Most importantly: I like the idea very much, but the paper clearly needs more work. There are broken citations and typos everywhere. I strongly suggest polishing this paper as it could be an important work in the model interpretability field.