From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper71/AnonReviewer1
title: Interesting, new methods for understanding LSTMs.
review: This paper presents a new perspective on LSTM interpretability and understanding. Since RNNs are so common in language tasks, it's a great fit for the workshop. Unlike previous studies of LSTMs and RNNs as dynamical systems (which often focus on observability and long-time behavior), this paper treats the dynamical system as a system, and perturbs it to quantify the system's behavior. While this is nothing new in systems and controls, this is an new application to LSTMS and has led to some insights (relationship of overfitting with settling time).

Some comments:
* The methodology can be more clear about what input is used for each measure. Is it the step input, sinusoidal input, or the baseline input signal?
* I would to see you allude to the argument given under definition 5 for studying individual LSTM cells earlier in the paper (like either in the introduction, or at the beginning of section 4) to better motivate studying individual cells. 
* The methods here are really interesting and potentially useful but the authors haven't yet shown its worth. For follow on work, I'd love to see the correlation between all measures. I'm curious if cells that have high correlation values also have high delta responses, or vise versa. This might look like figure 2c-g but with a plot of correlation values for cells, ordered by the delta response of the cell.
  - Particularly interesting are correlation and frequency, response output change and settling time, settling time and amplitude
* For a methods paper, I'd appreciate smaller experiments or more explanation that back up interpretations of each of the measures. 

This paper is so well written. Thank you for taking the time to edit and rewrite. Great features include the outline at the end of the introduction, detailed captions, and a great coverage of previous work in interpretability broken down by section. Keep up the great work!
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper71/AnonReviewer3
title: Some nice ideas but limited analysis and imprecise interpretation of results
review: The paper proposes a couple of methods for measuring various properties of the behavior of  LSTM cells. Most of the measurements are based on probing the network with periodic input signals and step functions and analyzing the reactions of the cells to these in terms of amplitude, frequency, settling time and correlations. Some initial analysis is done on a network trained on sequential MNIST. After that, networks are analysed after training on other types of sequential data. 

While the paper doesn’t investigate language or speech problems, it does deal with sequential data and interpretability so it probably fits within the scope of the workshop well enough. More methods for analyzing and interpreting neural network behavior are needed but I’m not sure that the ones presented in this paper are that new or insightful.

The empirical analysis is very informal and mostly consists of ad-hoc interpretation of the graphs of the measurements without much statistical rigor. Sometimes, the paper makes statements about the results that don’t seem to follow from the displayed plots at all. In Figure 4, it is not that clear that the grey line is a good predictor of the blue line at all. Section 5.3 contains statements like “the delta response predicted that neurons with high negative or positive value, are more significant in output’s prediction” where the meaning of ‘predicted’ is not clear. It would have been more insightful to see some correlations here with corresponding significance tests. The use of the term ‘significant’ in the section title doesn't seem to be based on any statistical meaning of that word. 

The authors claim that their analysis is more general in scope than for example Karpathy et al. (2015), but the nature of the input data (natural language) made that analysis at least more interpretable. Similarly, previous work on image networks benefits from the interpretability of visualizations of that type of data. Many of the measurements in this paper could be interesting depending on the data used, but are hard to interpret by themselves. I also do find the idea to use custom inputs to measure the responses of recurrent units interesting but am worried that those observations may not be very representative of the state dynamics displayed when the actual input is presented for which the network was trained. 

All in all, there are some interesting ideas in the paper, but their analysis is not well presented and seems very preliminary in nature, even for a workshop paper. Because of this, their value is hard to evaluate at this stage.

rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper71/AnonReviewer2
title: Review for response characterization for auditing cell dynamics in LSTMs.
review: In this short paper the authors suggest using traditional linear systems analysis to study the cells in the LSTM trained on a number of tasks.  The main idea is to pull the individual cell "out of the network" so to speak, and study it isolation according to 4 measures, based on inputting a step function or a sine wave into a given cell.  These are the setting time and and delta response for a step, or the output frequency and sine wave amplitude for an input sine wave.  The authors do this analysis for a few different toy tasks and show the distributions according to task.  The authors then go on to show that one may ablate individual units when sorted by these metrics and therefore indicate that these measures correlate to a minor/moderate degree with the loss and should therefore be considered useful.  Finally, the authors show these metrics sometimes change as a function of network size.

All in all, I think there is an interesting kernel in this paper but I did not really learn anything about how these networks functioned from studying the methods in this paper.  Perhaps larger examples, more compelling application of the analysis would improve it.

The paper is very well written, and easy to understand.

Small points

It would be useful to know exactly what equation is being analyzed, e.g. in Figure 2? I.e. what was the equation that was pulled out of the LSTM exactly?

Standard deviation of a moving average filter? What is the distribution over?
Fig 4C, 4D, 4G ref on page 7?




rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct