Original: Discovery of natural language concepts in individual units
Revised: Natural Language Detectors Emerge in Individual Neurons

Revised: Anonymous Author(s)

Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Although deep convolutional networks have achieved improved performance inmany natural language tasks, they have been treated as black boxes because they aredifﬁcult to interpret.
Revised: Although deep convolutional networks have achieved improved performance inmany natural language tasks, they have been treated as black boxes because they aredifﬁcult to interpret.

Original: Especially, little is known about how they represent languagein their intermediate layers.
Revised: Especially, little is known about how they represent languagein their intermediate layers.

Original: In an attempt to understand the representations ofdeep convolutional networks trained on language tasks, we show that individualunits are selectively responsive to speciﬁc morphemes, words, and phrases, ratherthan responding to arbitrary and uninterpretable patterns.
Revised: In an attempt to understand the representations ofdeep convolutional networks trained on language tasks, we show that individualunits are selectively responsive to speciﬁc morphemes, words, and phrases, ratherthan responding to arbitrary and uninterpretable patterns.

Original: In order to quantitativelyanalyze such intriguing phenomenon, we propose a concept alignment methodbased on how units respond to replicated text.
Revised: In order to quantitativelyanalyze such intriguing phenomenon, we propose a concept alignment methodbased on how units respond to replicated text.

Original: We conduct analyses with differentarchitectures on multiple datasets for classiﬁcation and translation tasks and providenew insights into how deep models understand natural language.
Revised: We conduct analyses with differentarchitectures on multiple datasets for classiﬁcation and translation tasks and providenew insights into how deep models understand natural language.

Original:  Introduction
Revised:  Introduction

Original: Understanding and interpreting how deep neural networks process natural language is a crucial andchallenging problem.
Revised: Understanding and interpreting how deep neural networks process natural language is a crucial andchallenging problem.

Original: While deep neural networks have achieved state-of-the-art performances inneural machine translation (NMT) [26, 6, 11, 29], sentiment classiﬁcation tasks [31, 7] and manymore, the sequence of non-linear transformations makes it difﬁcult for users to make sense of anypart of the whole model.
Revised: While deep neural networks have achieved state-of-the-art performances inneural machine translation (NMT) [26, 6, 11, 29], sentiment classiﬁcation tasks [31, 7] and manymore, the sequence of non-linear transformations makes it difﬁcult for users to make sense of anypart of the whole model.

Original: Because of their lack of interpretability, deep models are often regardedas hard to debug and unreliable for deployment, not to mention that they also prevent the user fromlearning about how to make better decisions based on the model’s outputs.
Revised: Because of their lack of interpretability, deep models are often regardedas hard to debug and unreliable for deployment, not to mention that they also prevent the user fromlearning about how to make better decisions based on the model’s outputs.

Original: An important research direction toward interpretable deep networks is to understand what their hiddenrepresentations learn and how they encode informative factors when solving the target task.
Revised: An important research direction toward interpretable deep networks is to understand what their hiddenrepresentations learn and how they encode informative factors when solving the target task.

Original: Amongthem, studies including Bau et al.
Revised: Amongthem, studies including Bau et al.

Original: [4], Fong & Vedaldi [10], Olah et al.
Revised: [4], Fong & Vedaldi [10], Olah et al.

Original: [20, 21] have researchedon what information is captured by individual or multiple units in visual representations learnedfor image recognition tasks.
Revised: [20, 21] have researchedon what information is captured by individual or multiple units in visual representations learnedfor image recognition tasks.

Original: These studies showed that some of the individual units are selectivelyresponsive to speciﬁc visual concepts, as opposed to getting activated in an uninterpretable manner.
Revised: These studies showed that some of the individual units are selectivelyresponsive to speciﬁc visual concepts, as opposed to getting activated in an uninterpretable manner.

Original: By analyzing individual units of deep networks, not only were they able to obtain more ﬁne-grainedinsights about the representations than analyzing representations as a whole, but they were alsoable to ﬁnd meaningful connections to various problems such as generalization of network [18] orgenerating explanations for the decision of the model [33, 21, 34].
Revised: By analyzing individual units of deep networks, not only were they able to obtain more ﬁne-grainedinsights about the representations than analyzing representations as a whole, but they were alsoable to ﬁnd meaningful connections to various problems such as generalization of network [18] orgenerating explanations for the decision of the model [33, 21, 34].

Original: Since these studies of unit-level representations have mainly been conducted on models learnedfor computer vision-oriented tasks, little is known about the representation of models learned fromnatural language processing (NLP) tasks.
Revised: Since these studies of unit-level representations have mainly been conducted on models learnedfor computer vision-oriented tasks, little is known about the representation of models learned fromnatural language processing (NLP) tasks.

Original: Several studies that have previously analyzed individualunits of natural language representations assumed that they align a predeﬁned set of speciﬁc concepts,such as sentiment present in the text [22], text lengths, quotes and brackets [12].
Revised: Several studies that have previously analyzed individualunits of natural language representations assumed that they align a predeﬁned set of speciﬁc concepts,such as sentiment present in the text [22], text lengths, quotes and brackets [12].

Original: They discovered theemergence of certain units that selectively activate to those speciﬁc concepts.
Revised: They discovered theemergence of certain units that selectively activate to those speciﬁc concepts.

Original: Building upon these
Revised: Building upon these

Original: [Figure]
Revised: [Figure]

Original: These rights
Revised: These rights

Original: [Figure]
Revised: [Figure]

Original: lines of research, we consider the following question: What natural language concepts are capturedby each unit in the representations learned from NLP tasks?
Revised: lines of research, we consider the following question: What natural language concepts are capturedby each unit in the representations learned from NLP tasks?

Original: To answer this question, we newly propose a simple but highly effective concept alignment methodthat can discover which natural language concepts are aligned to each unit in the representation.
Revised: To answer this question, we newly propose a simple but highly effective concept alignment methodthat can discover which natural language concepts are aligned to each unit in the representation.

Original: Herewe use the term unit to refer to each channel in convolutional representation, and natural languageconcepts to refer to the grammatical units of natural language that preserve meanings; i.e . morphemes,words, and phrases.
Revised: Herewe use the term unit to refer to each channel in convolutional representation, and natural languageconcepts to refer to the grammatical units of natural language that preserve meanings; i.e . morphemes,words, and phrases.

Original: Our approach ﬁrst identiﬁes the most activated sentences per unit and breaksthose sentences into these natural language concepts.
Revised: Our approach ﬁrst identiﬁes the most activated sentences per unit and breaksthose sentences into these natural language concepts.

Original: It then aligns speciﬁc concepts to each unit bymeasuring activation value of replicated text that indicates how much each concept contributes to theunit activation.
Revised: It then aligns speciﬁc concepts to each unit bymeasuring activation value of replicated text that indicates how much each concept contributes to theunit activation.

Original: This method also allows us to systematically analyze the concepts carried by units indiverse settings, including depth of layers, the form of supervision, and data-speciﬁc or task-speciﬁcdependencies.
Revised: This method also allows us to systematically analyze the concepts carried by units indiverse settings, including depth of layers, the form of supervision, and data-speciﬁc or task-speciﬁcdependencies.

Original: The contributions of this work can be summarized as follows:
Revised: The contributions of this work can be summarized as follows:

Original: • We show that the units of deep CNNs learned in NLP tasks could act as a natural languageconcept detector.
Revised: • We show that the units of deep CNNs learned in NLP tasks could act as a natural languageconcept detector.

Original: Without any additional labeled data or re-training process, we can discover,for each unit of the CNN, natural language concepts including morphemes, words andphrases that are present in the training
Revised: Without any additional labeled data or re-training process, we can discover,for each unit of the CNN, natural language concepts including morphemes, words andphrases that are present in the training

Original: data.•
Revised: data.•

Original: We systematically analyze what information is captured by units in representation in multiple
Revised: We systematically analyze what information is captured by units in representation in multiple

Original: settings by varying network architectures, tasks, and datasets.
Revised: settings by varying network architectures, tasks, and datasets.

Original: We use VDCNN [7] for
Revised: We use VDCNN [7] for

Original: sentiment and topic classiﬁcation tasks on Yelp Reviews, AG News [31], and DBpediaontology dataset [17] and ByteNet [11] for translation tasks on Europarl [16] and News
Revised: sentiment and topic classiﬁcation tasks on Yelp Reviews, AG News [31], and DBpediaontology dataset [17] and ByteNet [11] for translation tasks on Europarl [16] and News

Original: Commentary [28] datasets.
Revised: Commentary [28] datasets.

Original: • We also analyze how aligned natural language concepts evolve as the layer gets deeper.
Revised: • We also analyze how aligned natural language concepts evolve as the layer gets deeper.

Original: As part of our analysis, we show that our interpretation of learned representations couldbe utilized at designing network architectures with fewer parameters but with comparableperformance to baseline models.
Revised: As part of our analysis, we show that our interpretation of learned representations couldbe utilized at designing network architectures with fewer parameters but with comparableperformance to baseline models.

Original:  Related Work
Revised:  Related Work

Original: 2.1 Interpretation of Individual Units in Deep Models
Revised: 2.1 Interpretation of Individual Units in Deep Models

Original: Recent work on interpreting hidden representations at unit-level were mostly motivated from theircounterparts in computer vision.
Revised: Recent work on interpreting hidden representations at unit-level were mostly motivated from theircounterparts in computer vision.

Original: In computer vision community, Zhou et al.
Revised: In computer vision community, Zhou et al.

Original: [32] retrieved imagesamples with the highest unit activation, for each of units in a CNN trained on image recognitiontasks.
Revised: [32] retrieved imagesamples with the highest unit activation, for each of units in a CNN trained on image recognitiontasks.

Original: They used these retrieved samples to show that visual concepts like color, texture and objectparts are aligned to speciﬁc units, and the concepts were aligned to units by human annotators.
Revised: They used these retrieved samples to show that visual concepts like color, texture and objectparts are aligned to speciﬁc units, and the concepts were aligned to units by human annotators.

Original: Bauet al.
Revised: Bauet al.

Original: [4] introduced BRODEN dataset, which consists of pixel-level segmentation labels for diversevisual concepts and then analyzed the correlation between activation of each unit and such visualconcepts.
Revised: [4] introduced BRODEN dataset, which consists of pixel-level segmentation labels for diversevisual concepts and then analyzed the correlation between activation of each unit and such visualconcepts.

Original: In their work, although aligning concepts which absent from BRODEN dataset requiresadditional labeled images or human annotation, they showed that some individual units respond tospeciﬁc visual concepts.
Revised: In their work, although aligning concepts which absent from BRODEN dataset requiresadditional labeled images or human annotation, they showed that some individual units respond tospeciﬁc visual concepts.

Original: On the other hand, Erhan et al.
Revised: On the other hand, Erhan et al.

Original: [9], Olah et al.
Revised: [9], Olah et al.

Original: [20], Simonyan et al.
Revised: [20], Simonyan et al.

Original: [25] discovered visual conceptsaligned to each unit by optimizing a random initial image to maximize the unit activation by gradientdescent.
Revised: [25] discovered visual conceptsaligned to each unit by optimizing a random initial image to maximize the unit activation by gradientdescent.

Original: In these cases, the resulting interpretation of each unit is in the form of optimized images, andnot in the natural language form as the aforementioned ones.
Revised: In these cases, the resulting interpretation of each unit is in the form of optimized images, andnot in the natural language form as the aforementioned ones.

Original: However, these continuous interpretation results make it hard for further quantitative analyses of discrete properties of representations, suchas quantifying characteristics of representations in layer-wise [4] and correlations between theinterpretability of a unit and regularization [33].
Revised: However, these continuous interpretation results make it hard for further quantitative analyses of discrete properties of representations, suchas quantifying characteristics of representations in layer-wise [4] and correlations between theinterpretability of a unit and regularization [33].

Original: Nevertheless, these methods have the advantage thatthe results are not constrained to a predeﬁned set of concepts, giving ﬂexibility as to which conceptsare captured by each unit.
Revised: Nevertheless, these methods have the advantage thatthe results are not constrained to a predeﬁned set of concepts, giving ﬂexibility as to which conceptsare captured by each unit.

Original: In the NLP domain, studies including Karpathy et al.
Revised: In the NLP domain, studies including Karpathy et al.

Original: [12] and Tang et al.
Revised: [12] and Tang et al.

Original: [27] analyzed the internalmechanisms of deep models used for NLP and found intriguing properties that appear in units ofhidden representations.
Revised: [27] analyzed the internalmechanisms of deep models used for NLP and found intriguing properties that appear in units ofhidden representations.

Original: Among those studies, the closest one to ours is Radford et al.
Revised: Among those studies, the closest one to ours is Radford et al.

Original: [22], whodeﬁned a unit as each element in the representation of an LSTM learned for language modeling andfound that the concept of sentiment was aligned to a particular unit.
Revised: [22], whodeﬁned a unit as each element in the representation of an LSTM learned for language modeling andfound that the concept of sentiment was aligned to a particular unit.

Original: Compared with these previousstudies, we focus on discovering a much wider variety of natural language concepts , including anymorphemes, words, and phrases all found in the training data.
Revised: Compared with these previousstudies, we focus on discovering a much wider variety of natural language concepts , including anymorphemes, words, and phrases all found in the training data.

Original: To the best our knowledge, this isthe ﬁrst attempt to discover concepts among all that exist in the form of natural language from thetraining corpus.
Revised: To the best our knowledge, this isthe ﬁrst attempt to discover concepts among all that exist in the form of natural language from thetraining corpus.

Original: By extending the scope of detected concepts to meaningful building blocks of naturallanguage, we provide insights into how various linguistic features are encoded by the hidden units ofdeep representations.
Revised: By extending the scope of detected concepts to meaningful building blocks of naturallanguage, we provide insights into how various linguistic features are encoded by the hidden units ofdeep representations.

Original: 2.2 Analysis of deep representations learned for NLP tasks
Revised: 2.2 Analysis of deep representations learned for NLP tasks

Original: Most previous work that analyzes the learned representation of NLP tasks focused on constructingdownstream tasks that predict concepts of interest.
Revised: Most previous work that analyzes the learned representation of NLP tasks focused on constructingdownstream tasks that predict concepts of interest.

Original: A common approach is to measure the performanceof a regression/classiﬁcation model that predicts the concept of interest to see whether those conceptsare encoded in representation of a input sentence.
Revised: A common approach is to measure the performanceof a regression/classiﬁcation model that predicts the concept of interest to see whether those conceptsare encoded in representation of a input sentence.

Original: For example, Conneau et al.
Revised: For example, Conneau et al.

Original: [8], Adi et al.
Revised: [8], Adi et al.

Original: [2], Zhuet al.
Revised: [2], Zhuet al.

Original: [35] proposed several probing tasks to test whether the (non-)linear regression model can predictwell the syntactic or semantic information from the representation learned on translation tasks or theskip-thought or word embedding vectors.
Revised: [35] proposed several probing tasks to test whether the (non-)linear regression model can predictwell the syntactic or semantic information from the representation learned on translation tasks or theskip-thought or word embedding vectors.

Original: Shi et al.
Revised: Shi et al.

Original: [24], Belinkov et al.
Revised: [24], Belinkov et al.

Original: [5] constructed regressiontasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoderrepresentation of the learned model in translation task.
Revised: [5] constructed regressiontasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoderrepresentation of the learned model in translation task.

Original: Compared with previous work, our contributions can be summarized as follows.
Revised: Compared with previous work, our contributions can be summarized as follows.

Original: (1) By identifyingthe role of the individual units, rather than analyzing the representation as a whole, we provide moreﬁne-grained understanding of how the representations encode informative factors in training data.(2)
Revised: (1) By identifyingthe role of the individual units, rather than analyzing the representation as a whole, we provide moreﬁne-grained understanding of how the representations encode informative factors in training data.(2)

Original: Rather than limiting the linguistic features within the representation to be discovered, we focuson covering concepts of fundamental building blocks of natural language (morphemes, words, andphrases) present in the training data, providing more ﬂexible interpretation results without relying ona predeﬁned set of concepts.
Revised: Rather than limiting the linguistic features within the representation to be discovered, we focuson covering concepts of fundamental building blocks of natural language (morphemes, words, andphrases) present in the training data, providing more ﬂexible interpretation results without relying ona predeﬁned set of concepts.

Original: (3) Our concept alignment method does not need any additional labeleddata or re-training process, so it can always provide deterministic interpretation results using only thetraining data.
Revised: (3) Our concept alignment method does not need any additional labeleddata or re-training process, so it can always provide deterministic interpretation results using only thetraining data.

Original:  Approach
Revised:  Approach

Original: We focus on convolutional neural networks (CNNs), particularly their character-level variants.
Revised: We focus on convolutional neural networks (CNNs), particularly their character-level variants.

Original: CNNshave shown great success on various natural language applications, including translation, languagemodeling, and sentence classiﬁcation [11, 13, 31, 7].
Revised: CNNshave shown great success on various natural language applications, including translation, languagemodeling, and sentence classiﬁcation [11, 13, 31, 7].

Original: Compared to deep architectures based on fullyconnected layers, CNNs are natural candidates for unit-level analysis because their channel-levelrepresentations are reported to work as templates for detecting concepts [4].
Revised: Compared to deep architectures based on fullyconnected layers, CNNs are natural candidates for unit-level analysis because their channel-levelrepresentations are reported to work as templates for detecting concepts [4].

Original: Our approach for aligning natural language concepts to units is summarized as follows.
Revised: Our approach for aligning natural language concepts to units is summarized as follows.

Original: We ﬁrst train a
Revised: We ﬁrst train a

Original: CNN model for each natural language task and retrieve training sentences that highly activate speciﬁcunits.
Revised: CNN model for each natural language task and retrieve training sentences that highly activate speciﬁcunits.

Original: Interestingly, we discover morphemes, words, and phrases that appear dominantly within theseretrieved sentences, implying that those concepts have a signiﬁcant impact on the activation valueof the unit.
Revised: Interestingly, we discover morphemes, words, and phrases that appear dominantly within theseretrieved sentences, implying that those concepts have a signiﬁcant impact on the activation valueof the unit.

Original: Then, we ﬁnd a set of concepts which attribute a lot to the unit activation by measuringactivation value of each replicated candidate concept, and align them to unit.
Revised: Then, we ﬁnd a set of concepts which attribute a lot to the unit activation by measuringactivation value of each replicated candidate concept, and align them to unit.

Original: 3.1 The Model and The Task
Revised: 3.1 The Model and The Task

Original: We analyze representations learned on three classiﬁcation and four translation datasets shown in Table1.
Revised: We analyze representations learned on three classiﬁcation and four translation datasets shown in Table1.

Original: Training details for each dataset are available in Appendix B.
Revised: Training details for each dataset are available in Appendix B.

Original: We then focus on the representationsin each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al.
Revised: We then focus on the representationsin each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al.

Original: [Table]
Revised: [Table]

Original: pointed out, the representation of the decoder (the output layer in the case of classiﬁcation) isspecialized for predicting the output of the target task rather than for learning the semantics of theinput text.
Revised: pointed out, the representation of the decoder (the output layer in the case of classiﬁcation) isspecialized for predicting the output of the target task rather than for learning the semantics of theinput text.

Original: 3.2 Top K Activated Sentences Per Unit
Revised: 3.2 Top K Activated Sentences Per Unit

Original: Once we train a CNN model for a given task, we feed again all sentences in the training data to the
Revised: Once we train a CNN model for a given task, we feed again all sentences in the training data to the

Original: CNN and measure the activation in the unit of interest.
Revised: CNN and measure the activation in the unit of interest.

Original: The dimension of sentence representation isl × d , where l is the length of the activation map and d is the number of units per layer.
Revised: The dimension of sentence representation isl × d , where l is the length of the activation map and d is the number of units per layer.

Original: That is, theactivation of each of d units is l -dimensional.
Revised: That is, theactivation of each of d units is l -dimensional.

Original: For each unit, we retrieve top K training sentenceswith the highest mean activation over the l entries of the vector.
Revised: For each unit, we retrieve top K training sentenceswith the highest mean activation over the l entries of the vector.

Original: Interestingly, some natural languagepatterns such as morphemes, words, phrases frequently appear in the retrieved sentences, implyingthat those concepts might have a large attribution to the activation value of that unit.
Revised: Interestingly, some natural languagepatterns such as morphemes, words, phrases frequently appear in the retrieved sentences, implyingthat those concepts might have a large attribution to the activation value of that unit.

Original: 3.3 Concept Alignment with Replicated Text
Revised: 3.3 Concept Alignment with Replicated Text

Original: We propose a simple approach for identifying the concepts as follows.
Revised: We propose a simple approach for identifying the concepts as follows.

Original: For constructing candidateconcepts, we parse each of top K sentences with a constituency parser [15].
Revised: For constructing candidateconcepts, we parse each of top K sentences with a constituency parser [15].

Original: Within the constituencybased parse tree, we deﬁne candidate concepts as all terminal and non-terminal nodes ( e.g . fromsentence John hit the balls , we obtain candidate concepts as { John, hit, the, balls, the balls, hit theballs, John hit the balls } ).
Revised: Within the constituencybased parse tree, we deﬁne candidate concepts as all terminal and non-terminal nodes ( e.g . fromsentence John hit the balls , we obtain candidate concepts as { John, hit, the, balls, the balls, hit theballs, John hit the balls } ).

Original: We also break each word into morphemes using a morphological analysistool [30] and add them to candidate concepts ( e.g . from word balls , we obtain morphemes { ball,s } ).
Revised: We also break each word into morphemes using a morphological analysistool [30] and add them to candidate concepts ( e.g . from word balls , we obtain morphemes { ball,s } ).

Original: We repeat this process for every top K sentence and build a set of candidate concepts for unit u ,which is denoted as C u
Revised: We repeat this process for every top K sentence and build a set of candidate concepts for unit u ,which is denoted as C u

Original: = { c 1 , ..., c N } , where N is the number of candidate concepts of the unit.
Revised: = { c 1 , ..., c N } , where N is the number of candidate concepts of the unit.

Original: Next, we measure how each candidate concept attributes to the unit’s activation value.
Revised: Next, we measure how each candidate concept attributes to the unit’s activation value.

Original: We create asynthetic sentence by replicating each candidate concept so that its length is identical to the averagelength of all training sentences ( e.g . candidate concept the ball is replicated as the ball the ballthe ball... ).
Revised: We create asynthetic sentence by replicating each candidate concept so that its length is identical to the averagelength of all training sentences ( e.g . candidate concept the ball is replicated as the ball the ballthe ball... ).

Original: Replicated sentences are denoted as R = { r 1 , ..., r N } , and each r n ∈ R is forwardedto CNN, and their activation value of unit u is measured as a u ( r n ) ∈ R , which is averaged overl entries.
Revised: Replicated sentences are denoted as R = { r 1 , ..., r N } , and each r n ∈ R is forwardedto CNN, and their activation value of unit u is measured as a u ( r n ) ∈ R , which is averaged overl entries.

Original: Finally, the degree of alignment (DoA) between a candidate concept c n and a unit u isdeﬁned as follows:
Revised: Finally, the degree of alignment (DoA) between a candidate concept c n and a unit u isdeﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: In short, the DoA measures the extent to which unit
Revised: In short, the DoA measures the extent to which unit

Original: u ’s activation is sensitive to the presence ofcandidate concept c n .
Revised: u ’s activation is sensitive to the presence ofcandidate concept c n .

Original: If a candidate concept c n appears in the top K sentences and unit’s activationvalue a u is responsive to c n a lot, then DoA u,c n gets large, suggesting that candidate concept c n isstrongly aligned to unit u .
Revised: If a candidate concept c n appears in the top K sentences and unit’s activationvalue a u is responsive to c n a lot, then DoA u,c n gets large, suggesting that candidate concept c n isstrongly aligned to unit u .

Original: Finally, for each unit u , we deﬁne a set of its aligned concepts C ∗ u
Revised: Finally, for each unit u , we deﬁne a set of its aligned concepts C ∗ u

Original: = { c ∗ 1 , ..., c ∗ M } as M candidateconcepts with the largest DoA values in C u .
Revised: = { c ∗ 1 , ..., c ∗ M } as M candidateconcepts with the largest DoA values in C u .

Original: Depending on how we set M , we can detect differentnumbers of concepts per unit.
Revised: Depending on how we set M , we can detect differentnumbers of concepts per unit.

Original: In this experiments, we set M to 3.
Revised: In this experiments, we set M to 3.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Evaluation of concept alignment
Revised: 4.1 Evaluation of concept alignment

Original: To quantitatively evaluate how well our approach aligns concepts, we measure how selectivelyeach unit responds to the aligned concept.
Revised: To quantitatively evaluate how well our approach aligns concepts, we measure how selectivelyeach unit responds to the aligned concept.

Original: Motivated by Morcos et al.
Revised: Motivated by Morcos et al.

Original: [18], we deﬁne the concept
Revised: [18], we deﬁne the concept

Original: [Figure]
Revised: [Figure]

Original: S e l e c t i v i t
Revised: S e l e c t i v i t

Original: y replicate inclusion random signiﬁcantly selectivity of a unit u , to which a set of concepts C ∗ u that our alignment method detects, as follows:
Revised: y replicate inclusion random signiﬁcantly selectivity of a unit u , to which a set of concepts C ∗ u that our alignment method detects, as follows:

Original: [Equation]
Revised: [Equation]

Original: where S denotes all sentences in training set, and µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value ofunit activation when forwarding a set of sentences S + , which is deﬁned as one of the following:
Revised: where S denotes all sentences in training set, and µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value ofunit activation when forwarding a set of sentences S + , which is deﬁned as one of the following:

Original: • replicate : S
Revised: • replicate : S

Original: + contains the sentences created by replicating each concept in C ∗ u .
Revised: + contains the sentences created by replicating each concept in C ∗ u .

Original: As before,
Revised: As before,

Original: the sentence length is set as the average length of all training sentences for fair comparison.
Revised: the sentence length is set as the average length of all training sentences for fair comparison.

Original: • inclusion : S + contains the training sentences that include at least one concept in C ∗ u .
Revised: • inclusion : S + contains the training sentences that include at least one concept in C ∗ u .

Original: • random : S + contains randomly sampled sentences from the training data.
Revised: • random : S + contains randomly sampled sentences from the training data.

Original: In contrast, µ − = 1 |S − | (cid:80) s ∈S − a u ( s
Revised: In contrast, µ − = 1 |S − | (cid:80) s ∈S − a u ( s

Original: ) is the average value of unit activation when forwarding S − ,
Revised: ) is the average value of unit activation when forwarding S − ,

Original: which consists of sentences that do not include any concept in C ∗ u
Revised: which consists of sentences that do not include any concept in C ∗ u

Original: Intuitively, if unit u ’s activation is highly sensitive to C ∗ u ( i.e . those found by our alignment method)and if it is not to other factors, then Sel u gets large; otherwise, Sel u is near 0.
Revised: Intuitively, if unit u ’s activation is highly sensitive to C ∗ u ( i.e . those found by our alignment method)and if it is not to other factors, then Sel u gets large; otherwise, Sel u is near 0.

Original: Figure 2 shows the mean and variance of selectivity values for all units learned in each dataset forthe three S + categories.
Revised: Figure 2 shows the mean and variance of selectivity values for all units learned in each dataset forthe three S + categories.

Original: Consistent with our intuition, in all datasets, the mean selectivity of thereplicate set is the highest with a signiﬁcant margin, that of inclusion set is the runner-up, and that ofthe random set is the lowest.
Revised: Consistent with our intuition, in all datasets, the mean selectivity of thereplicate set is the highest with a signiﬁcant margin, that of inclusion set is the runner-up, and that ofthe random set is the lowest.

Original: These results support our claim that our method is successful to alignconcepts in which the unit responds selectively.
Revised: These results support our claim that our method is successful to alignconcepts in which the unit responds selectively.

Original: 4.2 Concept Alignment of Units
Revised: 4.2 Concept Alignment of Units

Original: Figure 3 shows examples of the top K sentences and the aligned concepts that are discovered byour method, for selected units.
Revised: Figure 3 shows examples of the top K sentences and the aligned concepts that are discovered byour method, for selected units.

Original: For each unit, we ﬁnd the top K = 10 sentences that activate themost in the several encoding layer of ByteNet and VDCNN, and select some of them (only up toﬁve sentences are shown due to space constraints).
Revised: For each unit, we ﬁnd the top K = 10 sentences that activate themost in the several encoding layer of ByteNet and VDCNN, and select some of them (only up toﬁve sentences are shown due to space constraints).

Original: We observe that some patterns appear frequentlywithin the top K sentences.
Revised: We observe that some patterns appear frequentlywithin the top K sentences.

Original: For example, in the top K sentences that activate unit 124 of 0th layerof ByteNet, the concepts of ‘(’, ‘)’, ‘-’ appear in common, while the concepts of soft, software, wiappear frequently in the sentences for unit 19 of 1st layer of VDCNN.
Revised: For example, in the top K sentences that activate unit 124 of 0th layerof ByteNet, the concepts of ‘(’, ‘)’, ‘-’ appear in common, while the concepts of soft, software, wiappear frequently in the sentences for unit 19 of 1st layer of VDCNN.

Original: These results qualitativelyshow that individual units are selectively responsive to speciﬁc natural language concepts.
Revised: These results qualitativelyshow that individual units are selectively responsive to speciﬁc natural language concepts.

Original: More interestingly, we discover that many units could capture speciﬁc meanings or syntactic rolesbeyond superﬁcial, low-level patterns.
Revised: More interestingly, we discover that many units could capture speciﬁc meanings or syntactic rolesbeyond superﬁcial, low-level patterns.

Original: For example, unit 690 of the 14th layer in ByteNet captures( what, who, where ) concepts, all of which play the similar grammatical role.
Revised: For example, unit 690 of the 14th layer in ByteNet captures( what, who, where ) concepts, all of which play the similar grammatical role.

Original: On the other hand, unit224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semanticallysimilar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge ( sure, know,aware ) and the VDCNN unit detecting years ( 1999, 1969, 1992 ).
Revised: On the other hand, unit224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semanticallysimilar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge ( sure, know,aware ) and the VDCNN unit detecting years ( 1999, 1969, 1992 ).

Original: This suggests that, although we traincharacter-level CNNs with feeding sentences as the form of discrete symbols ( i.e . character indices),individual units could capture natural language concepts sharing similar semantic or grammaticalrole.
Revised: This suggests that, although we traincharacter-level CNNs with feeding sentences as the form of discrete symbols ( i.e . character indices),individual units could capture natural language concepts sharing similar semantic or grammaticalrole.

Original: We note that there are units that detect concepts more abstract than just morphemes, words, or phrases,and for these units our method tends to align relevant lower-level concepts.
Revised: We note that there are units that detect concepts more abstract than just morphemes, words, or phrases,and for these units our method tends to align relevant lower-level concepts.

Original: For example, in units 477and 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top Ksentences, all top K sentences have similar nuances like positive and negative sentiments.
Revised: For example, in units 477and 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top Ksentences, all top K sentences have similar nuances like positive and negative sentiments.

Original: In thesecases, our method does capture relevant phrase-level concepts (e.g., very disappointing, absoluteworst place ), indicating that the higher-level nuance (e.g., negativity) is indirectly captured.
Revised: In thesecases, our method does capture relevant phrase-level concepts (e.g., very disappointing, absoluteworst place ), indicating that the higher-level nuance (e.g., negativity) is indirectly captured.

Original: • This is the first time, it is the first exercise.
Revised: • This is the first time, it is the first exercise.

Original: These, however, are just the first steps.
Revised: These, however, are just the first steps.

Original: • This ought to be the first step forward.
Revised: • This ought to be the first step forward.

Original: • That will be just the first step.
Revised: • That will be just the first step.

Original: • We can already see the first results.
Revised: • We can already see the first results.

Original: Layer 14,
Revised: Layer 14,

Original: [Table]
Revised: [Table]

Original: 360: the first step, first, be the first step • That is not the subject of this communication.
Revised: 360: the first step, first, be the first step • That is not the subject of this communication.

Original: • That is the purpose of this communication.
Revised: • That is the purpose of this communication.

Original: • I would like to ask the Commissioner for a reply.
Revised: • I would like to ask the Commissioner for a reply.

Original: • This is impossible without increasing efficiency.
Revised: • This is impossible without increasing efficiency.

Original: • Will we be able to achieve this, Commissioner?
Revised: • Will we be able to achieve this, Commissioner?

Original: [Table]
Revised: [Table]

Original: : of this communication, will, communication • qualcomm has inked a licensing agreement with Microsoft • peoplesoft wants its customers to get aggressive with software upgrades to increase efficiency.
Revised: : of this communication, will, communication • qualcomm has inked a licensing agreement with Microsoft • peoplesoft wants its customers to get aggressive with software upgrades to increase efficiency.

Original: • provide its customers with access to wi-fi hotspots around the world.
Revised: • provide its customers with access to wi-fi hotspots around the world.

Original: • realnetworks altered the software for market-leading ipod.
Revised: • realnetworks altered the software for market-leading ipod.

Original: [Table]
Revised: [Table]

Original: lost one war to microsoft by not licensing its mac…
Revised: lost one war to microsoft by not licensing its mac…

Original: [Table]
Revised: [Table]

Original: M o r ph e m e
Revised: M o r ph e m e

Original: W o r d
Revised: W o r d

Original: P
Revised: P

Original: h r a s e
Revised: h r a s e

Original: • Are you sure you are aware of our full potential?
Revised: • Are you sure you are aware of our full potential?

Original: • They know that
Revised: • They know that

Original: and we know that.
Revised: and we know that.

Original: • I am sure you will understand.
Revised: • I am sure you will understand.

Original: • I am sure you will do this.
Revised: • I am sure you will do this.

Original: • I am confident that we will find a solution.
Revised: • I am confident that we will find a solution.

Original: [Table]
Revised: [Table]

Original: know,
Revised: know,

Original: [Table]
Revised: [Table]

Original: • Who gets what, how much and when?
Revised: • Who gets what, how much and when?

Original: • On what basis, when and how?
Revised: • On what basis, when and how?

Original: Then we need to ask: where do we start?
Revised: Then we need to ask: where do we start?

Original: However, what should we do at this point?
Revised: However, what should we do at this point?

Original: • What I am wondering now is: where are they?
Revised: • What I am wondering now is: where are they?

Original: [Table]
Revised: [Table]

Original: • [COM
Revised: • [COM

Original: [Table]
Revised: [Table]

Original: 24 - C
Revised: 24 - C

Original: [Table]
Revised: [Table]

Original: 0527/2001 - 2001/2207(COS)] • Exemptions will follow a two-stage procedure.
Revised: 0527/2001 - 2001/2207(COS)] • Exemptions will follow a two-stage procedure.

Original: • Such exceptions were completely inappropriate.
Revised: • Such exceptions were completely inappropriate.

Original: • (Exchange of views with microphones switched off) • [COM(2001) 1 - C5-0007/2001 - 2001/0005(COD)]')
Revised: • (Exchange of views with microphones switched off) • [COM(2001) 1 - C5-0007/2001 - 2001/0005(COD)]')

Original: • google has a new one for the labs - google suggest • google has released google desktop
Revised: • google has a new one for the labs - google suggest • google has released google desktop

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: disappointing
Revised: disappointing

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: the hell did i pay for
Revised: the hell did i pay for

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: place
Revised: place

Original: [Table]
Revised: [Table]

Original: have ever
Revised: have ever

Original: [Table]
Revised: [Table]

Original: business with!
Revised: business with!

Original: [Table]
Revised: [Table]

Original: the is by
Revised: the is by

Original: [Table]
Revised: [Table]

Original: the worst restaurant i have ever been to…
Revised: the worst restaurant i have ever been to…

Original: [Table]
Revised: [Table]

Original: place
Revised: place

Original: [Table]
Revised: [Table]

Original: a rip off!
Revised: a rip off!

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We also note that, because the number of morphemes, words and phrases present in training corpus isusually much greater than the number of units per layer, we do not expect to always align any naturallanguage concepts in the corpus to one of the units.
Revised: We also note that, because the number of morphemes, words and phrases present in training corpus isusually much greater than the number of units per layer, we do not expect to always align any naturallanguage concepts in the corpus to one of the units.

Original: Our approach thus tends to ﬁnd concepts that areconsidered as more important than others for solving the target task.
Revised: Our approach thus tends to ﬁnd concepts that areconsidered as more important than others for solving the target task.

Original: Overall, these results suggest how input sentences are represented in the hidden representation of the
Revised: Overall, these results suggest how input sentences are represented in the hidden representation of the

Original: CNN as follows:
Revised: CNN as follows:

Original: • Several units in the CNN learned on NLP tasks respond selectively to speciﬁc natural
Revised: • Several units in the CNN learned on NLP tasks respond selectively to speciﬁc natural

Original: language concepts, rather than getting activated in an uninterpretable way.
Revised: language concepts, rather than getting activated in an uninterpretable way.

Original: This
Revised: This

Original: means that
Revised: means that

Original: these units can serve as detectors for speciﬁc natural language concepts.•
Revised: these units can serve as detectors for speciﬁc natural language concepts.•

Original: There are units capturing syntactically or semantically related concepts, suggesting that
Revised: There are units capturing syntactically or semantically related concepts, suggesting that

Original: they model the meaning or grammatical role shared between those concepts , as opposed tosuperﬁcially modeling each natural language symbol .
Revised: they model the meaning or grammatical role shared between those concepts , as opposed tosuperﬁcially modeling each natural language symbol .

Original: 4.3 Concept Distribution in Layers
Revised: 4.3 Concept Distribution in Layers

Original: Using the concept alignments found earlier, we can visualize how concepts are distributed acrosslayers.
Revised: Using the concept alignments found earlier, we can visualize how concepts are distributed acrosslayers.

Original: Figure 4 shows the concepts of the units in the 0th, 1st, 2nd, 3rd layer of VDCNN learnedon AG-News dataset, and 0th, 5th, 9th and 14th layer of the ByteNet encoder learned on English- to-German Europarl dataset with their number of aliged units.
Revised: Figure 4 shows the concepts of the units in the 0th, 1st, 2nd, 3rd layer of VDCNN learnedon AG-News dataset, and 0th, 5th, 9th and 14th layer of the ByteNet encoder learned on English- to-German Europarl dataset with their number of aliged units.

Original: For each layer, we sort concepts ina decreasing order by the number of aligned units and show 30 concepts most aligned.
Revised: For each layer, we sort concepts ina decreasing order by the number of aligned units and show 30 concepts most aligned.

Original: Recall that,since we align concepts for each unit, there are concepts aligned to multiple units simultaneously.
Revised: Recall that,since we align concepts for each unit, there are concepts aligned to multiple units simultaneously.

Original: Concept distribution for other datasets are available in appendix C.
Revised: Concept distribution for other datasets are available in appendix C.

Original: Overall, we ﬁnd that data and task-speciﬁc concepts are likely to be aligned to many units.
Revised: Overall, we ﬁnd that data and task-speciﬁc concepts are likely to be aligned to many units.

Original: In AG
Revised: In AG

Original: News, since the task is to classify given sentences into following categories; World , Sports , Businessand Science/Tech , concepts related to these topics commonly emerge.
Revised: News, since the task is to classify given sentences into following categories; World , Sports , Businessand Science/Tech , concepts related to these topics commonly emerge.

Original: Similarly, we can see that unitslearned for Europarl dataset focus to encode some key words in the training corpus.
Revised: Similarly, we can see that unitslearned for Europarl dataset focus to encode some key words in the training corpus.

Original: 4.4 How does concept granularity evolve with layer?
Revised: 4.4 How does concept granularity evolve with layer?

Original: In computer vision tasks, visual concepts captured by units in CNN representations learned for imagerecognition tasks evolve with layer depths; color, texture concepts are emergent in earlier layers andmore abstract concepts like parts and objects are emergent in deeper layers.
Revised: In computer vision tasks, visual concepts captured by units in CNN representations learned for imagerecognition tasks evolve with layer depths; color, texture concepts are emergent in earlier layers andmore abstract concepts like parts and objects are emergent in deeper layers.

Original: To conﬁrm that it alsoholds for representations learned in NLP tasks, we divide granularity of natural language concepts tomorhpeme, word and N -gram phrase ( N = 2 , 3 , 4 , 5 ), and observe the number of units that they arealigned in different layers.
Revised: To conﬁrm that it alsoholds for representations learned in NLP tasks, we divide granularity of natural language concepts tomorhpeme, word and N -gram phrase ( N = 2 , 3 , 4 , 5 ), and observe the number of units that they arealigned in different layers.

Original: Figure 5 shows this trend, where in lower layers such as the 0th layer, less phrase concepts but moremorphemes and words are detected.
Revised: Figure 5 shows this trend, where in lower layers such as the 0th layer, less phrase concepts but moremorphemes and words are detected.

Original: This is because we use a character-level CNN, whose receptiveﬁelds of convolution may not be large enough to detect lengthy phrases.
Revised: This is because we use a character-level CNN, whose receptiveﬁelds of convolution may not be large enough to detect lengthy phrases.

Original: Further, interestingly intranslation cases, we observe that aligned concepts signiﬁcantly change in shallower layers ( e.g . fromthe 0th to the 4th), but do not change much from middle to deeper layers ( e.g . from the 5th to the14th).
Revised: Further, interestingly intranslation cases, we observe that aligned concepts signiﬁcantly change in shallower layers ( e.g . fromthe 0th to the 4th), but do not change much from middle to deeper layers ( e.g . from the 5th to the14th).

Original: Thus, it remains for us to answer the following question: for the representations learned on translationdatasets, why does concept granularity not evolve much in deeper layers?
Revised: Thus, it remains for us to answer the following question: for the representations learned on translationdatasets, why does concept granularity not evolve much in deeper layers?

Original: One possibility is thatthe capacity of the network is large enough so that the representations in middle layers could besufﬁciently informative to solve the task.
Revised: One possibility is thatthe capacity of the network is large enough so that the representations in middle layers could besufﬁciently informative to solve the task.

Original: To validate this hypothesis, we re-train ByteNet fromscratch while varying only layer depth of the encoder and ﬁxing other conditions.
Revised: To validate this hypothesis, we re-train ByteNet fromscratch while varying only layer depth of the encoder and ﬁxing other conditions.

Original: We record their
Revised: We record their

Original: BLEU scores on the validation data as shown in Figure 6.
Revised: BLEU scores on the validation data as shown in Figure 6.

Original: The performance of the translation modeldoes not change much with more than six encoder layers, but it signiﬁcantly drops at the models withfewer than 4 encoder layers.
Revised: The performance of the translation modeldoes not change much with more than six encoder layers, but it signiﬁcantly drops at the models withfewer than 4 encoder layers.

Original: This trend coincides with the result from Figure 5 that the evolution ofconcept granularity stops around middle-to-higher layers.
Revised: This trend coincides with the result from Figure 5 that the evolution ofconcept granularity stops around middle-to-higher layers.

Original: This shared pattern suggests that about sixencoder layers are enough to encode informative factors in the given datasets to perform optimally on
Revised: This shared pattern suggests that about sixencoder layers are enough to encode informative factors in the given datasets to perform optimally on

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: by varying the number of encoder layers.
Revised: by varying the number of encoder layers.

Original: the translation task.
Revised: the translation task.

Original: In deeper models, this may suggest that the middle layer’s representation may bealready informative enough to encode the input text, and our result may partly coincide with that of
Revised: In deeper models, this may suggest that the middle layer’s representation may bealready informative enough to encode the input text, and our result may partly coincide with that of

Original: Mou et al.
Revised: Mou et al.

Original: [19], which shows that representation of intermediate layers is more transferable than thatof deeper layers in language tasks, unlike in computer vision where deeper layers are usually moreuseful and discriminative.
Revised: [19], which shows that representation of intermediate layers is more transferable than thatof deeper layers in language tasks, unlike in computer vision where deeper layers are usually moreuseful and discriminative.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a simple but highly effective concept alignment method for character-level CNNs toconﬁrm that each unit of the hidden layers serves as detectors of natural language concepts.
Revised: We proposed a simple but highly effective concept alignment method for character-level CNNs toconﬁrm that each unit of the hidden layers serves as detectors of natural language concepts.

Original: Usingthis method, we analyzed the characteristics of units with multiple datasets on classiﬁcation andtranslation tasks.
Revised: Usingthis method, we analyzed the characteristics of units with multiple datasets on classiﬁcation andtranslation tasks.

Original: Consequently, we shed light on how deep representations capture natural language,and how they vary with various conditions.
Revised: Consequently, we shed light on how deep representations capture natural language,and how they vary with various conditions.

Original: An interesting future direction is to extend the concept coverage from natural language to moreabstract forms such as sentence structure, nuance and tone.
Revised: An interesting future direction is to extend the concept coverage from natural language to moreabstract forms such as sentence structure, nuance and tone.

Original: Another direction is to quantify theproperties of individual units in other models widely used in NLP tasks.
Revised: Another direction is to quantify theproperties of individual units in other models widely used in NLP tasks.

Original: In particular, combining ourdeﬁnition of concepts with the attention mechanism ( e.g .
Revised: In particular, combining ourdeﬁnition of concepts with the attention mechanism ( e.g .

Original: Bahdanau et al.
Revised: Bahdanau et al.

Original: [3]) could be a promisingdirection, because it can reveal how the representations are attended by the model to capture concepts,helping us better understand the decision making process of popular deep models.
Revised: [3]) could be a promisingdirection, because it can reveal how the representations are attended by the model to capture concepts,helping us better understand the decision making process of popular deep models.

Original: A Other metrics for DoA with biased alignment result
Revised: A Other metrics for DoA with biased alignment result

Original: In section 3.3, we deﬁne Degree of Alignment (DoA) between concept c n and unit u as activationvalue of unit u for replication of c n .
Revised: In section 3.3, we deﬁne Degree of Alignment (DoA) between concept c n and unit u as activationvalue of unit u for replication of c n .

Original: We tried lots of stuff while we were working on DoA metrics,but a lot of it give biased concept alignment result for several reasons.
Revised: We tried lots of stuff while we were working on DoA metrics,but a lot of it give biased concept alignment result for several reasons.

Original: We here provide the things wetried and their reasons of failure.
Revised: We here provide the things wetried and their reasons of failure.

Original: A.1 Point-wise Mutual Information
Revised: A.1 Point-wise Mutual Information

Original: Point-wise Mutual Information (PMI) is a measure of association used in information theory andstatistics.
Revised: Point-wise Mutual Information (PMI) is a measure of association used in information theory andstatistics.

Original: The PMI of a pair of samples x and y sampled from random variables X and Y quantiﬁesthe discrepancy between the probability of their coincidence as follows:
Revised: The PMI of a pair of samples x and y sampled from random variables X and Y quantiﬁesthe discrepancy between the probability of their coincidence as follows:

Original: [Equation]
Revised: [Equation]

Original: We then deﬁne DoA between candidate concept c n and unit u by using PMI as follow:
Revised: We then deﬁne DoA between candidate concept c n and unit u by using PMI as follow:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: However, this metric has bias of always preferring lengthy concepts even in earlier layers, which isnot possible considering the receptive ﬁeld of the convolution.
Revised: However, this metric has bias of always preferring lengthy concepts even in earlier layers, which isnot possible considering the receptive ﬁeld of the convolution.

Original: Our intuition for this bias is consistentwith Role & Nadif [23],where it is a well-known problem with PMI, which is its tendency to givevery high association scores to pairs involving low-frequency ones, as the denominator is small insuch cases.
Revised: Our intuition for this bias is consistentwith Role & Nadif [23],where it is a well-known problem with PMI, which is its tendency to givevery high association scores to pairs involving low-frequency ones, as the denominator is small insuch cases.

Original: If certain concept c n in top K sentences is very lengthy, then its frequency in the corpusp ( c n ) would get very small, and pmi( u , c n ) would be large with regardless of correlation between uand c n .
Revised: If certain concept c n in top K sentences is very lengthy, then its frequency in the corpusp ( c n ) would get very small, and pmi( u , c n ) would be large with regardless of correlation between uand c n .

Original: A.2 Concept Occlusion
Revised: A.2 Concept Occlusion

Original: We tested concept alignments with following concept occlusion method.
Revised: We tested concept alignments with following concept occlusion method.

Original: For each of top K sentences,we replace a c n by dummy character tokens which have no meaning, forward it to the model, andmeasure the reduction of the unit activation value.
Revised: For each of top K sentences,we replace a c n by dummy character tokens which have no meaning, forward it to the model, andmeasure the reduction of the unit activation value.

Original: We repeat this for every candidate concept in thesentences – as a result, we can identify which candidate concept greatly reduce unit activation values.
Revised: We repeat this for every candidate concept in thesentences – as a result, we can identify which candidate concept greatly reduce unit activation values.

Original: We thus deﬁne concepts aligned to each unit as the candidate concept that consistently lower the unitactivation across the top K sentences.
Revised: We thus deﬁne concepts aligned to each unit as the candidate concept that consistently lower the unitactivation across the top K sentences.

Original: More formally, for each unit u , let S = { s 1 , ..., s K } be top K activated sentences.
Revised: More formally, for each unit u , let S = { s 1 , ..., s K } be top K activated sentences.

Original: Since weocclude each candidate concept in sentences, we deﬁne the set of candidate concept C = { c 1 , ..., c N } ,obtained from parsing each sentence in S .
Revised: Since weocclude each candidate concept in sentences, we deﬁne the set of candidate concept C = { c 1 , ..., c N } ,obtained from parsing each sentence in S .

Original: We deﬁne the degree of alignment (DoA) between a concept c ∈ C and a unit u as:
Revised: We deﬁne the degree of alignment (DoA) between a concept c ∈ C and a unit u as:

Original: [Equation]
Revised: [Equation]

Original: where Z is a normalizing factor, and a u indicates the mean activation of unit u , Occ c n ( s ) is a sentences where candidate concept c n is occluded, and 1 ( c n ∈ s ) is an indicator of whether c n is included inthe sentence s .
Revised: where Z is a normalizing factor, and a u indicates the mean activation of unit u , Occ c n ( s ) is a sentences where candidate concept c n is occluded, and 1 ( c n ∈ s ) is an indicator of whether c n is included inthe sentence s .

Original: In short, the DoA measures how much candidate concept contributes to the activationof the unit’s top K sentences.
Revised: In short, the DoA measures how much candidate concept contributes to the activationof the unit’s top K sentences.

Original: If a candidate concept c n appears in the top K sentences S and greatlyreduces the activation of unit u , then DoA u,c n gets large, implying that the c n is strongly aligned tounit u .
Revised: If a candidate concept c n appears in the top K sentences S and greatlyreduces the activation of unit u , then DoA u,c n gets large, implying that the c n is strongly aligned tounit u .

Original: Unfortunately, this metric could not fairly compare attribution of several candidate concepts.
Revised: Unfortunately, this metric could not fairly compare attribution of several candidate concepts.

Original: Forexample, consider following two concepts c 1 = hit , c 2 = hit the ball are included in one sentence.
Revised: Forexample, consider following two concepts c 1 = hit , c 2 = hit the ball are included in one sentence.

Original: Occluding c 2 might gives relatively large decrement in unit activation value than that of c 1 , since c 1includes c 2 .
Revised: Occluding c 2 might gives relatively large decrement in unit activation value than that of c 1 , since c 1includes c 2 .

Original: For this reason, occlusion based metric is unnecessarily dependant of length of concept,rather than it’s attribution.
Revised: For this reason, occlusion based metric is unnecessarily dependant of length of concept,rather than it’s attribution.

Original: A.3 Inclusion Selectivity
Revised: A.3 Inclusion Selectivity

Original: Note that inclusion selectivity in section 4.1 is also used as DoA. Recall that inclusion selectivity iscalculated as equation 2.
Revised: Note that inclusion selectivity in section 4.1 is also used as DoA. Recall that inclusion selectivity iscalculated as equation 2.

Original: In this case, µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value of unit activationwhen forwarding a set of sentences S
Revised: In this case, µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value of unit activationwhen forwarding a set of sentences S

Original: + , where S + denotes that sentences including candidate conceptc n .
Revised: + , where S + denotes that sentences including candidate conceptc n .

Original: However, it induces similar bias which is similar to section A.1.
Revised: However, it induces similar bias which is similar to section A.1.

Original: It always prefers lengthy phrasessince those lengthy concept occur few times in entire corpus.
Revised: It always prefers lengthy phrasessince those lengthy concept occur few times in entire corpus.

Original: For example, assume that activationvalue of unit u for the sentence including speciﬁc lengthy phase is very high.
Revised: For example, assume that activationvalue of unit u for the sentence including speciﬁc lengthy phase is very high.

Original: If such phrase occuronly one time over the entire corpus, µ + is equal to the activation value of the sentence, which isrelatively very high than µ + for other candidate concepts.
Revised: If such phrase occuronly one time over the entire corpus, µ + is equal to the activation value of the sentence, which isrelatively very high than µ + for other candidate concepts.

Original: This error could be alleviated on verylarge corpus where every candidate concept occurs enough in the corpus so that estimation of µ + getrelatively accurate, which is practically not possible.
Revised: This error could be alleviated on verylarge corpus where every candidate concept occurs enough in the corpus so that estimation of µ + getrelatively accurate, which is practically not possible.

Original: B Training Details
Revised: B Training Details

Original: In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classiﬁcation tasks,both to analyze properties of representations for language.
Revised: In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classiﬁcation tasks,both to analyze properties of representations for language.

Original: Training details are as follows.
Revised: Training details are as follows.

Original: B.1 ByteNet
Revised: B.1 ByteNet

Original: [Figure]
Revised: [Figure]

Original: We trained a ByteNet on the translation tasks, in particular on the WMT’17 English-to-German
Revised: We trained a ByteNet on the translation tasks, in particular on the WMT’17 English-to-German

Original: Europarl dataset, the English-to-German news dataset, WMT’16 English-to-French, English-to-Czechnews dataset.
Revised: Europarl dataset, the English-to-German news dataset, WMT’16 English-to-French, English-to-Czechnews dataset.

Original: We used the same model architecture and hyperparameters for both datasets.
Revised: We used the same model architecture and hyperparameters for both datasets.

Original: We setthe batch size to 8 and the learning rate to 0.001.
Revised: We setthe batch size to 8 and the learning rate to 0.001.

Original: The parameters were optimized with Adam [14]for 5 epochs, and early stopping was actively used for ﬁnding parameters that generalize well.
Revised: The parameters were optimized with Adam [14]for 5 epochs, and early stopping was actively used for ﬁnding parameters that generalize well.

Original: Ourcode is based on a TensorFlow [1] implementation of ByteNet found in https://github.com/paarthneekhara/byteNet-tensorflow .
Revised: Ourcode is based on a TensorFlow [1] implementation of ByteNet found in https://github.com/paarthneekhara/byteNet-tensorflow .

Original: B.2 Very Deep CNN (VDCNN)
Revised: B.2 Very Deep CNN (VDCNN)

Original: We trained a VDCNN for classﬁcation tasks, in particular on the AG News dataset, the binarizedversion of the Yelp Reviews dataset, and DBpedia ontology dataset.
Revised: We trained a VDCNN for classﬁcation tasks, in particular on the AG News dataset, the binarizedversion of the Yelp Reviews dataset, and DBpedia ontology dataset.

Original: For each task, we used 1 temporalconvolutional layer, 4 convolutional blocks with each convolutional layer having a ﬁlter width of 3.
Revised: For each task, we used 1 temporalconvolutional layer, 4 convolutional blocks with each convolutional layer having a ﬁlter width of 3.

Original: In our experiments, we analyze representations of each convolutional block layer.
Revised: In our experiments, we analyze representations of each convolutional block layer.

Original: The number ofunits in each layer representation is 64, 128, 256, 512 respectively.
Revised: The number ofunits in each layer representation is 64, 128, 256, 512 respectively.

Original: We set the batch size to 64 andthe learning rate to 0.01.
Revised: We set the batch size to 64 andthe learning rate to 0.01.

Original: The parameters are optimized using SGD optimizer for 50 epochs, and earlystopping is actively used.
Revised: The parameters are optimized using SGD optimizer for 50 epochs, and earlystopping is actively used.

Original: For each of the AG News, Yelp Reviews and DBpedia datasets, a VDCNNwas learned with the same structure and hyperparameters.
Revised: For each of the AG News, Yelp Reviews and DBpedia datasets, a VDCNNwas learned with the same structure and hyperparameters.

Original: Our code is based on a TensorFlowimplementation of VDCNN found in https://github.com/zonetrooper32/VDCNN .
Revised: Our code is based on a TensorFlowimplementation of VDCNN found in https://github.com/zonetrooper32/VDCNN .

Original: C Concept Distribution in layers for other datasets
Revised: C Concept Distribution in layers for other datasets

Original: In section 4.3, we visualized how concepts are distributed across layers, where model is trained on
Revised: In section 4.3, we visualized how concepts are distributed across layers, where model is trained on

Original: AG News dataset and English-to-German Europarl dataset.
Revised: AG News dataset and English-to-German Europarl dataset.

Original: Here, Figure 7 shows concept distributionin other datasets noted in Table 1.
Revised: Here, Figure 7 shows concept distributionin other datasets noted in Table 1.

Original: In the classiﬁcation tasks, we expect to ﬁnd more concepts that are directly related to predictingthe output label, as opposed to the translation tasks where the representations may have to includeinformation on most of the words for an accurate translation.
Revised: In the classiﬁcation tasks, we expect to ﬁnd more concepts that are directly related to predictingthe output label, as opposed to the translation tasks where the representations may have to includeinformation on most of the words for an accurate translation.

Original: While our goal is not to relate eachconcept to one of labels, we ﬁnd several concepts that are more predictive to a particular label thanothers.
Revised: While our goal is not to relate eachconcept to one of labels, we ﬁnd several concepts that are more predictive to a particular label thanothers.

Original: In consistent with section 4.3, there are data-speciﬁc and task-speciﬁc concepts aligned in each layer;i.e .
Revised: In consistent with section 4.3, there are data-speciﬁc and task-speciﬁc concepts aligned in each layer;i.e .

Original: { worst, 2 stars, awful } at Yelp Review, { ﬁlm, ship, school } at DBpedia, and some key wordsat translation datasets.
Revised: { worst, 2 stars, awful } at Yelp Review, { ﬁlm, ship, school } at DBpedia, and some key wordsat translation datasets.

Original: Note that Yelp Review and DBpedia is a classiﬁcation dataset, where modelis required to predict the polarity ( i.e . +1 or -1) or ontology ( i.e . Company, Educational Institution,
Revised: Note that Yelp Review and DBpedia is a classiﬁcation dataset, where modelis required to predict the polarity ( i.e . +1 or -1) or ontology ( i.e . Company, Educational Institution,

Original: Artist, Athelete, Ofﬁceholder, Mean of Transportation, Building, Natural Place, Village, Animal,
Revised: Artist, Athelete, Ofﬁceholder, Mean of Transportation, Building, Natural Place, Village, Animal,

Original: Plant, Album, Film, Written Work) for given sentence in supervised setting.
Revised: Plant, Album, Film, Written Work) for given sentence in supervised setting.