From: NIPS.cc/2018/Workshop/IRASL/Paper21/Authors
title: Extended paper is accepted at ICLR 2019
comment: 
Extended version of this work is accepted at ICLR 2019. Please refer to the extended paper at below link.

https://openreview.net/forum?id=S1EERs09YQ

From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Reject

From: NIPS.cc/2018/Workshop/IRASL/Paper21/Authors
title: Reply for Reviewer 2
comment: Thank you for your valuable comment. 

We also think that it would be interesting to see a similar analysis of deep convolutional models applied to spoken language, especially in tasks such as sentiment analysis in speech where prosody is an important concept. Our current work does involve only written text, as it has been our primary focus in this project. Nevertheless, because our approach on written text led to some interesting interpretations of the channels of deep CNNs, we’re also inclined to believe that an analogous analysis on popular deep convolutional speech models such as WaveNet could lead to the discovery of some interesting spoken-language units.

From: NIPS.cc/2018/Workshop/IRASL/Paper21/Authors
title: Reply for Reviewer 3
comment: We thank Reviewer 3 for your constructive review. Below, we respond to each comment in detail.

A-(1) K value in Figure 2
============================================================
Thanks for pointing out the unclearness of our formulation. At Section 4.1, we set K to 10 equally in Section 4.2. We will add this specification in the camera-ready version.

A-(2) Replicated sentences
============================================================
(Please also see A-(2) in our reply for Reviewer 1, who we think also pointed out a similar issue.) 
The primary reason why we replicate candidate concepts is to normalize the "degree of signal affecting the unit activation". It is an important consideration when comparing DoA values between candidate concepts in Section 3.3. We don't think that normalizing the sentence length with pad tokens would work as suggested, because of the compositional nature of words (from morphemes) and phrases (from words) that are also considered as concepts in our work. For example, suppose we have three candidate concepts "A", "B", "A B" with their padded version "A <pad>" "B <pad>" "A B". We also suppose that the unit is sensitive to two concepts "A" and "B". In this case, since the concept "A B" consists of the composition of two other concepts and both concepts affect the activation of the unit, DoA value of "A B" would be the highest. Thus, in most cases, the DoA metric in this way has a bias to prefer lengthy phrase concept. On the other hand, replicate sentences such as "A B A B ... A B" can avoid this issue while also having the advantages stated in our response A-(2) to Reviewer 1. (It does also introduce an extra "B A" 2-gram, but the convolutional filters are designed to filter patterns that matter "A B" rather than meaningless ones "B A").

From: NIPS.cc/2018/Workshop/IRASL/Paper21/Authors
title: Reply for Reviewer 1
comment: Thank you for your constructive review. Below, we respond to each comment in detail.

A-(1) Quantitative results for alignment
============================================================
As suggested, we can quantitatively define the interpretability of each unit as follows. Suppose that, for all concepts aligned to a unit, we obtain the unit activations by constructing replicate sentences with each concept (which is equal to $\mathcal{R}$). We can then say that a unit is not interpretable if all values in $\mathcal{R}$ are lower than the activation of the unit's top activated sentence. In this case, it is difficult to regard a unit as sensitive to the concept itself, because a replicate sentence consisting only of aligned concept has less impact on that unit than the top activated sentence. In this way, we can classify each unit on whether it is interpretable or not, and then measure the ratio of the interpretable units of each layer. This experiment shows that the ratio of interpretable units gets larger as the layer gets deeper. We will add this result to the camera ready version.


A-(2) Replicated sentences
============================================================
(i) Since the sensitivity of replicated sentences is much higher than that of inclusion and random as shown in Section 4.1, the replicate setting can reveal whether or not the unit has a role of natural language concept detector. Our replicate method for maximizing unit activation was motivated in a study of interpretability community of computer vision [1]. In Olah et al. [1], a random initial image is updated to maximize the activation of specific unit of pretrained CNN, and then certain visual concept appears repeatedly in the image. Motivated from this result, we assumed that unit activation also increased when we replicated the natural language concept, and experimentally it was validated in Section 4.1.

(ii) It is a good suggestion to normalize length of replicated sentences with different value. We experimentally confirmed that unit activation is proportional to the length of replicated aligned concepts and will add it to the camera ready version. However, it should be also noted that the comparisons with other selectivity values of inclusion, random can be fair by normalizing the length of the replicated sentence to the average length of the training sentences.

(iii) The replicate setting is also proposed for fair comparison between DoA values of each candidate concept. Instead of replicating candidate concept, if we obtain DoA value by constructing sentence with just one instance of the candidate concept (that is, "ball" rather than "ball ball ball..."), we add unintended a bias in comparing DoA values.
For example, suppose we have three candidate concepts "A", "B", "A C B" and the unit is sensitive to two concepts "A" and "B". In this case, since the concept "A C B" consists of the composition of two other concepts and both concepts affect the activation of the unit, DoA value of "A C B" would be the highest. Thus, in most cases, the DoA metric in this way has a bias to prefer lengthy phrase concept. Empirically, we also find that the average length of aligned concepts at the 0-th layer of translation model is 18.70 when we do not consider replicate setting. This is an unreasonable result considering that the receptive field of the convolution for 0-th layer is 5 in our model. To minimize this bias, we replicate each concept to the same length. In this case, the average length of concepts in 0-th layer is 6.99.


Reference
============================================================
[1] Chris Olah et al, Feature Visualization (Distill 2017), url: https://distill.pub/2017/feature-visualization/

From: NIPS.cc/2018/Workshop/IRASL/Paper21/AnonReviewer1
title: Simple and insightful procedure.
review: The paper describes a way to interpret hidden unit activation in a network architecture with respect to natural language concepts in different NLP tasks. With the term concept, the paper intends grammatical units of natural language like morphemes, words and phrases. 

The approach can be described as follows:
1. first, a network is trained for a given task using original sentences;
2. for each unit, the K sentences for which the unit presents the higher activation are selected;
3. from each of these sentences, all the concepts are extracted exploiting the constituency tree and breaking words into morphemes;
4. so called replicated sentences are synthetically generated using only these concepts;
5. the activation of each unit is measured using these replicated sentences, some random sentences from the training set that have at least one candidate concept for the unit, and some random sentences from the training set;
6. the selectivity metric is used, measuring how much a unit is sensitive to a certain input;
6. As expected, the highest average selectivity is achieved with replicated sentences, while the lowest is achieved with the random sentences, bringing evidence of the fact that that particular unit is particularly sensitive to the selected candidate topics. Moreover, since the input is fed char-by-char, this shows that the units are still capable to capture language concepts spanning across several characters.
7. Further evaluation is intended to assess the distribution of the concepts through layers.

Technically speaking, I think that the presented approach is sound. In particular, the selectivity metric and the evaluation presented until section 4.1 has the merit of being simple and intuitive and looking robust and motivated. The further evaluation looks still quite sound. In particular. In particular, one issue I found in Section 4.2 lines 207-210 and "Several" at line 213: why not providing some numbers? How many concepts do not activate any unit? How many unit are not activated by any concept in particular?---in particular, I cannot find in the paper any statement that specifies if all the units have been considered. This would be an approximate but still quantifiable measure of the portion of knowledge of the network that cannot be interpreted.

Another issue is with the construction of the replicated sentences. Intuitively, their role is quite clear especially after having read Section 4.1, though I think that their introduction should come with some motivation. Moreover, why having them only of the average sentence length? Has any other distribution been experimented?

In general the paper is also well written, though an extensive proof reading should be performed before the publication.
Other minor issues:
-lines 121-123 are repeated at 137-139;
- line 268, I don't see the point of having 'e.g.' in italics.

Overall, I think that the paper should be accepted for publication.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper21/AnonReviewer3
title: Simple yet interesting procedure
review: The authors describe a simple yet interesting procedure to interpret hidden unit activations in several deep neural network architectures for natural language processing tasks. The authors clearly show that some hidden units have the tendency to selectively activate when presented with some low-level natural language concepts as morphemes or words. The selectivity of hidden units with respect to higher-level concepts as sentiment or nuance is instead investigated only qualitatively. The review of the related works seems sufficient, and the advancements of the presented method are clearly stated. 

My biggest source of concern is a lack of formality which may leave the reader confused with respect to some relevant points of the discussion and impact the reproducibility of the work. For example, the authors do not provide explicitly the value of K used to build Figure 2. Also, I would have liked a more formal connection between the activation map of dimension l introduced in section 3.2 and the typical feature map of CNN layers.

Additionally, even if the procedure used to measure the degree of alignment between concept and unit is clear, the authors do not explicitly motivate why they build a set of replicated sentences. My guess is that  replicating candidate concepts is somehow a way of normalizing the degree of alignment with respect to the sentence length, but if that is the case then a clarification of the effects of the sentence length on the degree of alignment or on the concept selectivity would have been necessary. I guess a source of confusion from my side is that in the original VDCNN paper the input to the network is an embedding of padded text of fixed size, thus I will not expect the activations to be dependent on the sentence length.

The writing is good enough to allow the reader to understand the discussion, but in general the paper would largely benefit from a meticulous review of grammar and writing style.

In conclusion, I found the work interesting and I would recommend it for publication after extensive proofreading and a clarification of the points discussed above. 
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper21/AnonReviewer2
title: Discovery of natural language concepts in individual units
review: The paper defines term unit to refer to each channel in convolutional representation with CNNs, and natural language concepts to refer to the grammatical units of natural language that preserve meanings; i.e. morphemes, words, and phrases to show that the units of deep CNNs learned in NLP tasks could act as a natural language concept detector. 
Topic classification and translation tasks are considered only in written text.

The major problem in this paper is that it does not consider any aspect of the spoken language. For example, prosody should be considered as an important element in the analysis of sentiments in spoken languages. 
There are other relevant signal aspects that should be considered when concepts are hypothesized in continuous speech. 




rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct