Original: D ISCOVERY OF NATURAL LANGUAGE CONCEPTS IN INDIVIDUAL UNITS Anonymous authors
Revised: D ISCOVERY OF N ATURAL L ANGUAGE C ONCEPTS IN I NDIVIDUAL U NITS OF CNN S

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difﬁcult to interpret.
Revised: Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difﬁcult to interpret.

Original: Especially, little is known about how they represent language in their intermediate layers.
Revised: Especially, little is known about how they represent language in their intermediate layers.

Original: In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to speciﬁc morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns.
Revised: In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to speciﬁc morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns.

Original: In order to quantitatively analyze such  intriguing phenomenon, we propose a concept alignment method based on how units respond to  replicated text.
Revised: In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text.

Original: We conduct analyses with different architectures on multiple datasets for classiﬁcation and translation tasks and provide new insights into how deep models understand natural language.
Revised: We conduct analyses with different architectures on multiple datasets for classiﬁcation and translation tasks and provide new insights into how deep models understand natural language.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding and interpreting how deep neural networks process natural language is a crucial and challenging problem.
Revised: Understanding and interpreting how deep neural networks process natural language is a crucial and challenging problem.

Original: While deep neural networks have achieved state-of-the-art performances in neural machine translation (NMT) (Sutskever et al., 2014; Cho et al., 2014; Kalchbrenner et al., 2016; Vaswani et al., 2017), sentiment classiﬁcation tasks (Zhang et al., 2015; Conneau et al., 2017) and many more, the sequence of non-linear transformations makes it difﬁcult for users to make sense of any part of the whole model.
Revised: While deep neural networks have achieved state-of-the-art performances in neural machine translation (NMT) (Sutskever et al., 2014; Cho et al., 2014; Kalchbrenner et al., 2016; Vaswani et al., 2017), sentiment classiﬁcation tasks (Zhang et al., 2015; Conneau et al., 2017) and many more, the sequence of non-linear transformations makes it difﬁcult for users to make sense of any part of the whole model.

Original: Because of their lack of interpretability, deep models are often regarded as hard to debug and unreliable for deployment, not to mention that they also prevent the user from learning about how to make better decisions based on the model’s outputs.
Revised: Because of their lack of interpretability, deep models are often regarded as hard to debug and unreliable for deployment, not to mention that they also prevent the user from learning about how to make better decisions based on the model’s outputs.

Original: An important research direction toward interpretable deep networks is to understand what their hidden representations learn and how they encode informative factors when solving the target task.
Revised: An important research direction toward interpretable deep networks is to understand what their hidden representations learn and how they encode informative factors when solving the target task.

Original: Some studies including Bau et al.
Revised: Some studies including Bau et al.

Original: Fong & Vedaldi (2018); Olah et al.
Revised: Fong & Vedaldi (2018); Olah et al.

Original: (2017; 2018) have researched on what information is captured by individual or multiple units in visual representations learned for image recognition tasks.
Revised: (2017; 2018) have researched on what information is captured by individual or multiple units in visual representations learned for image recognition tasks.

Original: These studies showed that some of the individual units are selectively responsive to speciﬁc visual concepts, as opposed to getting activated in an uninterpretable manner.
Revised: These studies showed that some of the individual units are selectively responsive to speciﬁc visual concepts, as opposed to getting activated in an uninterpretable manner.

Original: By analyzing individual units of deep networks, not only were they able to obtain more ﬁne-grained insights about the representations than analyzing representations as a whole, but they were also able to ﬁnd meaningful connections to various problems such as generalization of network (Morcos et al., 2018) or generating explanations for the decision of the model (Zhou et al., 2018a;
Revised: By analyzing individual units of deep networks, not only were they able to obtain more ﬁne-grained insights about the representations than analyzing representations as a whole, but they were also able to ﬁnd meaningful connections to various problems such as generalization of network (Morcos et al., 2018), generating explanations for the decision of the model (Zhou et al., 2018a;

Original: Olah et al., 2018;
Revised: Olah et al., 2018;

Original: Zhou et al., 2018b).
Revised: Zhou et al., 2018b) and controlling the output of generative model (Bau et al., 2019).

Original: Since these studies of unit-level representations have mainly been conducted on models learned for computer vision-oriented tasks, little is known about the representation of models learned from natural language processing (NLP) tasks.
Revised: Since these studies of unit-level representations have mainly been conducted on models learned for computer vision-oriented tasks, little is known about the representation of models learned from natural language processing (NLP) tasks.

Original: Several studies that have previously analyzed individual units of natural language representations assumed that they align a predeﬁned set of speciﬁc concepts, such as sentiment present in the text (Radford et al., 2017), text lengths, quotes and brackets (Karpathy et al., 2015).
Revised: Several studies that have previously analyzed individual units of natural language representations assumed that they align a predeﬁned set of speciﬁc concepts, such as sentiment present in the text (Radford et al., 2017), text lengths, quotes and brackets (Karpathy et al., 2015).

Original: They discovered the emergence of certain units that selectively activate to those speciﬁc concepts.
Revised: They discovered the emergence of certain units that selectively activate to those speciﬁc concepts.

Original: Building upon these lines of research, we consider the following question: What natural language concepts are captured by each unit in the representations learned from NLP tasks?
Revised: Building upon these lines of research, we consider the following question: What natural language concepts are captured by each unit in the representations learned from NLP tasks?

Revised: [Figure]

Original: To answer this question, we newly propose a simple but highly effective concept alignment method that can discover which natural language concepts are aligned to each unit in the representation.
Revised: To answer this question, we newly propose a simple but highly effective concept alignment method that can discover which natural language concepts are aligned to each unit in the representation.

Original: Here we use the term unit to refer to each channel in convolutional representation, and natural language concepts to refer to the grammatical units of natural language that preserve meanings; i.e .[Figure] morphemes, words, and phrases.
Revised: Here we use the term unit to refer to each channel in convolutional representation, and natural language concepts to refer to the grammatical units of natural language that preserve meanings; i.e . morphemes, words, and phrases.

Original: Our approach ﬁrst identiﬁes the most activated sentences per unit and breaks those sentences into these natural language concepts.
Revised: Our approach ﬁrst identiﬁes the most activated sentences per unit and breaks those sentences into these natural language concepts.

Original: It then aligns speciﬁc concepts to each unit by measuring activation value of replicated text that indicates how much each concept contributes to the unit activation.
Revised: It then aligns speciﬁc concepts to each unit by measuring activation value of replicated text that indicates how much each concept contributes to the unit activation.

Original: This method also allows us to systematically analyze the concepts carried by units in diverse settings, including depth of layers, the form of supervision, and dataspeciﬁc or task-speciﬁc dependencies.
Revised: This method also allows us to systematically analyze the concepts carried by units in diverse settings, including depth of layers, the form of supervision, and dataspeciﬁc or task-speciﬁc dependencies.

Original: The contributions of this work can be summarized as follows:
Revised: The contributions of this work can be summarized as follows:

Original: • We show that the units of deep CNNs learned in NLP tasks could act as a natural language concept detector.
Revised: • We show that the units of deep CNNs learned in NLP tasks could act as a natural language concept detector.

Original: Without any additional labeled data or re-training process, we can discover, for each unit of the CNN, natural language concepts including morphemes, words and phrases that are present in the training data.
Revised: Without any additional labeled data or re-training process, we can discover, for each unit of the CNN, natural language concepts including morphemes, words and phrases that are present in the training data.

Original: • We systematically analyze what information is captured by units in representation in multiple settings by varying network architectures, tasks, and datasets.
Revised: • We systematically analyze what information is captured by units in representation across multiple settings by varying network architectures, tasks, and datasets.

Original: We use VDCNN (Conneau et al., 2017) for sentiment and topic classiﬁcation tasks on Yelp Reviews, AG News (Zhang et al., 2015), and DBpedia ontology dataset (Lehmann et al., 2015) and ByteNet (Kalchbrenner et al., 2016) for translation tasks on Europarl (Koehn, 2005) and News Commentary (Tiedemann, 2012) datasets.
Revised: We use VDCNN (Conneau et al., 2017) for sentiment and topic classiﬁcation tasks on Yelp Reviews, AG News (Zhang et al., 2015), and DBpedia ontology dataset (Lehmann et al., 2015) and ByteNet (Kalchbrenner et al., 2016) for translation tasks on Europarl (Koehn, 2005) and News Commentary (Tiedemann, 2012) datasets.

Original: • We also analyze how aligned natural language concepts evolve as the layer gets deeper.
Revised: • We also analyze how aligned natural language concepts evolve as they get represented in deeper layers.

Original: As part of our analysis, we show that our interpretation of learned representations could be utilized at designing network architectures with fewer parameters but with comparable performance to baseline models.
Revised: As part of our analysis, we show that our interpretation of learned representations could be utilized at designing network architectures with fewer parameters but with comparable performance to baseline models.

Original:  R ELATED W ORK 2. I NTERPRETATION OF I NDIVIDUAL U NITS IN D EEP M ODELS
Revised:  R ELATED W ORK 2. I NTERPRETATION OF I NDIVIDUAL U NITS IN D EEP M ODELS

Original: Recent work on interpreting hidden representations at unit-level were mostly motivated from their counterparts in computer vision.
Revised: Recent works on interpreting hidden representations at unit-level were mostly motivated by their counterparts in computer vision.

Original: In the computer vision community, Zhou et al.
Revised: In the computer vision community, Zhou et al.

Original: (2015) retrieved image samples with the highest unit activation, for each of units in a CNN trained on image recognition tasks.
Revised: (2015) retrieved image samples with the highest unit activation, for each of units in a CNN trained on image recognition tasks.

Original: They used these retrieved samples to show that visual concepts like color, texture and object parts are aligned to speciﬁc units, and the concepts were aligned to units by human annotators.
Revised: They used these retrieved samples to show that visual concepts like color, texture and object parts are aligned to speciﬁc units, and the concepts were aligned to units by human annotators.

Original: Bau et al.
Revised: Bau et al.

Original: (2017) introduced BRODEN dataset, which consists of pixel-level segmentation labels for diverse visual concepts and then analyzed the correlation between activation of each unit and such visual concepts.
Revised: (2017) introduced BRODEN dataset, which consists of pixel-level segmentation labels for diverse visual concepts and then analyzed the correlation between activation of each unit and such visual concepts.

Original: In their work, although aligning concepts which absent from BRODEN dataset requires additional labeled images or human annotation, they  showed that some individual units respond to speciﬁc visual concepts.
Revised: In their work, although aligning concepts which absent from BRODEN dataset requires additional labeled images or human annotation, they quantitatively showed that some individual units respond to speciﬁc visual concepts.

Original: On the other hand, Erhan et al.
Revised: On the other hand, Erhan et al.

Original: ; Olah et al.
Revised: ; Olah et al.

Original: ; Simonyan et al.
Revised: ; Simonyan et al.

Original: (2013) discovered visual concepts aligned to each unit by optimizing a random initial image to maximize the unit activation by gradient descent.
Revised: (2013) discovered visual concepts aligned to each unit by optimizing a random initial image to maximize the unit activation by gradient descent.

Original: In these cases, the resulting interpretation of each unit is in the form of optimized images, and not in the natural language form as the aforementioned ones.
Revised: In these cases, the resulting interpretation of each unit is in the form of optimized images, and not in the natural language form as the aforementioned ones.

Original: However, these continuous interpretation results make it hard for further quantitative analyses of discrete properties of representations, such as quantifying characteristics of representations with layer depth (Bau et al., 2017) and correlations between the interpretability of a unit and regularization (Zhou et al., 2018a).
Revised: However, these continuous interpretation results make it hard for further quantitative analyses of discrete properties of representations, such as quantifying characteristics of representations with layer depth (Bau et al., 2017) and correlations between the interpretability of a unit and regularization (Zhou et al., 2018a).

Original: Nevertheless, these methods have the advantage that the results are not constrained to a predeﬁned set of concepts, giving ﬂexibility as to which concepts are captured by each unit.[Table]
Revised: Nevertheless, these methods have the advantage that the results are not constrained to a predeﬁned set of concepts, giving ﬂexibility as to which concepts are captured by each unit.

Original: In the NLP domain, studies including Karpathy et al.
Revised: In the NLP domain, studies including Karpathy et al.

Original: ; Tang et al.
Revised: ; Tang et al.

Original: ; Qian et al.
Revised: ; Qian et al.

Original: Shi et al.
Revised: Shi et al.

Original: (2016a) analyzed the internal mechanisms of deep models used for NLP and found intriguing properties that appear in units of hidden representations.
Revised: (2016a) analyzed the internal mechanisms of deep models used for NLP and found intriguing properties that appear in units of hidden representations.

Original: Among those studies, the closest one to ours is Radford et al.
Revised: Among those studies, the closest one to ours is Radford et al.

Original: (2017), who deﬁned a unit as each element in the representation of an
Revised: (2017), who deﬁned a unit as each element in the representation of an

Original: LSTM learned for language modeling and found that the concept of sentiment was aligned to a particular unit.
Revised: LSTM learned for language modeling and found that the concept of sentiment was aligned to a particular unit.

Original: Compared with these previous studies, we focus on discovering a much wider variety of natural language concepts , including any morphemes, words, and phrases all found in the training data.
Revised: Compared with these previous studies, we focus on discovering a much wider variety of natural language concepts , including any morphemes, words, and phrases all found in the training data.

Original: To the best our knowledge, this is the ﬁrst attempt to discover concepts among all that exist in the form of natural language from the training corpus.
Revised: To the best our knowledge, this is the ﬁrst attempt to discover concepts among all that exist in the form of natural language from the training corpus.

Original: By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations.
Revised: By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations.

Original: 2. A NALYSIS OF D EEP R EPRESENTATIONS L EARNED FOR NLP T ASKS
Revised: 2. A NALYSIS OF D EEP R EPRESENTATIONS L EARNED FOR NLP T ASKS

Original: Most previous work that analyzes the learned representation of NLP tasks focused on constructing downstream tasks that predict concepts of interest.
Revised: Most previous work that analyzes the learned representation of NLP tasks focused on constructing downstream tasks that predict concepts of interest.

Original: A common approach is to measure the performance of a classiﬁcation model that predicts the concept of interest to see whether those concepts are encoded in representation of a input sentence.
Revised: A common approach is to measure the performance of a classiﬁcation model that predicts the concept of interest to see whether those concepts are encoded in representation of a input sentence.

Original: For example, Conneau et al.
Revised: For example, Conneau et al.

Original: ; Adi et al.
Revised: ; Adi et al.

Original: (2017); Zhu et al.
Revised: (2017); Zhu et al.

Original: (2018) proposed several probing tasks to test whether the (non-)linear regression model can predict well the syntactic or semantic information from the representation learned on translation tasks or the skip-thought or word embedding vectors.
Revised: (2018) proposed several probing tasks to test whether the (non-)linear regression model can predict well the syntactic or semantic information from the representation learned on translation tasks or the skip-thought or word embedding vectors.

Original: Shi et al.
Revised: Shi et al.

Original: (2016b)
Revised: (2016b)

Original: ; Belinkov et al.
Revised: ; Belinkov et al.

Original: (2017) constructed regression tasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoder representation of the learned model in translation task.
Revised: (2017) constructed regression tasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoder representation of the learned model in translation task.

Original: Compared with previous work, our contributions can be summarized as follows.
Revised: Compared with previous work, our contributions can be summarized as follows.

Original: (1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more ﬁne-grained understanding of how the representations encode informative factors in training data.
Revised: (1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more ﬁne-grained understanding of how the representations encode informative factors in training data.

Original: (2) Rather than limiting the linguistic features within the representation to be discovered, we focus on covering concepts of fundamental building blocks of natural language (morphemes, words, and phrases) present in the training data, providing more ﬂexible interpretation results without relying on a predeﬁned set of concepts.
Revised: (2) Rather than limiting the linguistic features within the representation to be discovered, we focus on covering concepts of fundamental building blocks of natural language (morphemes, words, and phrases) present in the training data, providing more ﬂexible interpretation results without relying on a predeﬁned set of concepts.

Original: (3) Our concept alignment method does not need any additional labeled data or re-training process, so it can always provide deterministic interpretation results using only the training data.
Revised: (3) Our concept alignment method does not need any additional labeled data or re-training process, so it can always provide deterministic interpretation results using only the training data.

Original:  A PPROACH
Revised:  A PPROACH

Original: We focus on convolutional neural networks (CNNs), particularly their character-level variants.
Revised: We focus on convolutional neural networks (CNNs), particularly their character-level variants.

Original: CNNs have shown great success on various natural language applications, including translation, language modeling, and sentence classiﬁcation (Kalchbrenner et al., 2016;
Revised: CNNs have shown great success on various natural language applications, including translation and sentence classiﬁcation (Kalchbrenner et al., 2016;

Original: Kim et al., 2016; Zhang et al., 2015; Conneau et al., 2017).
Revised: Kim et al., 2016; Zhang et al., 2015; Conneau et al., 2017).

Original: Compared to deep architectures based on fully connected layers, CNNs are natural candidates for unit-level analysis because their channel-level representations are reported to work as templates for detecting concepts (Bau et al., 2017).
Revised: Compared to deep architectures based on fully connected layers, CNNs are natural candidates for unit-level analysis because their channel-level representations are reported to work as templates for detecting concepts (Bau et al., 2017).

Original: Our approach for aligning natural language concepts to units is summarized as follows.
Revised: Our approach for aligning natural language concepts to units is summarized as follows.

Original: We ﬁrst train a CNN model for each natural language task  and retrieve training sentences that highly activate speciﬁc units.
Revised: We ﬁrst train a CNN model for each natural language task ( e.g . translation and classiﬁcation) and retrieve training sentences that highly activate speciﬁc units.

Original: Interestingly, we discover morphemes, words, and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a signiﬁcant impact on the activation value of the unit.
Revised: Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a signiﬁcant impact on the activation value of the unit.

Original: Then, we ﬁnd a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.
Revised: Then, we ﬁnd a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Original: 3. T HE M ODEL AND T HE T ASK

Original: We analyze representations learned on three classiﬁcation and four translation datasets shown in

Original: Table 1.

Original: Training details for each dataset are available in Appendix B.

Original: We then focus on the representations in each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al.

Original: (2016) pointed out, the representation of the decoder (the output layer in the case of classiﬁcation) is specialized for predicting the output of the target task rather than for learning the semantics of the input text.

Original: 3. T OP K A CTIVATED S ENTENCES P ER U NIT
Revised: 3. T OP K A CTIVATED S ENTENCES P ER U NIT

Original: Once we train a CNN model for a given task, we feed again all sentences S in the training set to the CNN model and record their activations.
Revised: Once we train a CNN model for a given task, we feed again all sentences S in the training set to the CNN model and record their activations.

Original: Given a layer and sentence s ∈ S , let A lu ( s )  denote the activation of unit u at spatial location l .
Revised: Given a layer and sentence s ∈ S , let A lu ( s ) [Table] denote the activation of unit u at spatial location l .

Original: Then, for unit u , we average activations over all spatial locations as a u ( s )
Revised: Then, for unit u , we average activations over all spatial locations as a u ( s )

Original: = 1 Z (cid:80)
Revised: = 1 Z (cid:80)

Original: l A lu ( s ) , where Z is a normalizer.
Revised: l A lu ( s ) , where Z is a normalizer.

Original: We then retrieve top K training sentences per unit with the highest mean activation a u .
Revised: We then retrieve top K training sentences per unit with the highest mean activation a u .

Original: Interestingly, some natural language patterns such as morphemes, words, phrases frequently appear in the retrieved sentences, implying that those concepts might have a large attribution to the activation value of that unit.
Revised: Interestingly, some natural language patterns such as morphemes, words, phrases frequently appear in the retrieved sentences (see Figure 1), implying that those concepts might have a large attribution to the activation value of that unit.

Original: 3. C ONCEPT A LIGNMENT WITH R EPLICATED T EXT
Revised: 3. C ONCEPT A LIGNMENT WITH R EPLICATED T EXT

Original: We propose a simple approach for identifying the concepts as follows.
Revised: We propose a simple approach for identifying the concepts as follows.

Original: For constructing candidate concepts, we parse each of top K sentences with a constituency parser (Kitaev & Klein, 2018).
Revised: For constructing candidate concepts, we parse each of top K sentences with a constituency parser (Kitaev & Klein, 2018).

Original: Within the constituency-based parse tree, we deﬁne candidate concepts as all terminal and nonterminal nodes ( e.g . from sentence John hit the balls , we obtain candidate concepts as { John, hit, the, balls, the balls, hit the balls, John hit the balls } ).
Revised: Within the constituency-based parse tree, we deﬁne candidate concepts as all terminal and nonterminal nodes ( e.g . from sentence John hit the balls , we obtain candidate concepts as { John, hit, the, balls, the balls, hit the balls, John hit the balls } ).

Original: We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts ( e.g . from word balls , we obtain morphemes { ball, s } ).
Revised: We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts ( e.g . from word balls , we obtain morphemes { ball, s } ).

Original: We repeat this process for every top K sentence and build a set of candidate concepts for unit u , which is denoted as C u
Revised: We repeat this process for every top K sentence and build a set of candidate concepts for unit u , which is denoted as C u

Original: = { c 1 , ..., c N } , where N is the number of candidate concepts of the unit.
Revised: = { c 1 , ..., c N } , where N is the number of candidate concepts of the unit.

Original: Next, we measure how each candidate concept attributes to the unit’s activation value.
Revised: Next, we measure how each candidate concept contributes to the unit’s activation value.

Original: For normalizing the degree of an input signal to the unit activation, we create a synthetic sentence by replicating each candidate concept so that its length is identical to the average length of all training sentences ( e.g . candidate concept the ball is replicated as the ball the ball the ball... ).
Revised: For normalizing the degree of an input signal to the unit activation, we create a synthetic sentence by replicating each candidate concept so that its length is identical to the average length of all training sentences ( e.g . candidate concept the ball is replicated as the ball the ball the ball... ).

Original: Replicated sentences are denoted as R = { r 1 , ..., r N } , and each r n ∈ R is forwarded to CNN, and their activation value of unit u is measured as a u ( r n ) ∈ R , which is averaged over l entries.
Revised: Replicated sentences are denoted as R = { r 1 , ..., r N } , and each r n ∈ R is forwarded to CNN, and their activation value of unit u is measured as a u ( r n ) ∈ R , which is averaged over l entries.

Original: Finally, the degree of alignment (DoA) between a candidate concept c n and a unit u is deﬁned as follows:
Revised: Finally, the degree of alignment (DoA) between a candidate concept c n and a unit u is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: In short, the DoA 1 measures the extent to unit u ’s activation is sensitive to the presence of candidate concept c n .
Revised: In short, the DoA 1 measures the extent to unit u ’s activation is sensitive to the presence of candidate concept c n .

Original: If a candidate concept c n appears in the top K sentences and unit’s activation value a u is responsive to c n a lot, then DoA u,c n gets large, suggesting that candidate concept c n is strongly aligned to unit u .
Revised: If a candidate concept c n appears in the top K sentences and unit’s activation value a u is responsive to c n a lot, then DoA u,c n gets large, suggesting that candidate concept c n is strongly aligned to unit u .

Original: Finally, for each unit u , we deﬁne a set of its aligned concepts C ∗ u
Revised: Finally, for each unit u , we deﬁne a set of its aligned concepts C ∗ u

Original: = { c ∗ 1 , ..., c ∗ M } as M candidate concepts with the largest DoA values in C u .
Revised: = { c ∗ 1 , ..., c ∗ M } as M candidate concepts with the largest DoA values in C u .

Original: Depending on how we set M , we can detect different numbers of concepts per unit.
Revised: Depending on how we set M , we can detect different numbers of concepts per unit.

Original: In this experiments, we set M to 3.
Revised: In this experiment, we set M to 3.

Original:  E XPERIMENTS 4. E VALUATION OF CONCEPT ALIGNMENT
Revised:  E XPERIMENTS 4. T HE M ODEL AND T HE T ASK

Revised: We analyze representations learned on three classiﬁcation and four translation datasets shown in Table 1. Training details for each dataset are available in Appendix B.

Revised: We then focus on the representations in each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al.

Revised: (2016) pointed out, the representation of the decoder (the output layer in the case of classiﬁcation) is specialized for predicting the output of the target task rather than for learning the semantics of the input text.

Revised: en-de-news en-fr-news en-cs-news en-de-europarl ag yelp dbpedia

Revised: S e l e c t i v i

Revised: t y replicate one instance inclusion random

Revised: 4. E VALUATION OF CONCEPT ALIGNMENT

Original: To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept.
Revised: To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept.

Original: Motivated by Morcos et al.
Revised: Motivated by Morcos et al.

Original: (2018), we deﬁne the concept selectivity of a unit u , towhich a set of concepts C ∗ u that our alignment method detects, as follows:
Revised: (2018), we deﬁne the concept selectivity of a unit u , to a set of concepts C ∗ u that our alignment method detects, as follows:

Original: [Equation]
Revised: [Equation]

Original: en-de-news en-fr-news en-cs-news en-de-europarl ag yelp dbpedia

Original: S e l e c t i v i t y replicate one instance inclusion random where S denotes all sentences in training set, and µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value of unit activation when forwarding a set of sentences S + , which is deﬁned as one of the following:
Revised: where S denotes all sentences in training set, and µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value of unit activation when forwarding a set of sentences S + , which is deﬁned as one of the following:

Original: • replicate : S
Revised: • replicate : S

Original: + contains the sentences created by replicating each concept in C ∗ u .
Revised: + contains the sentences created by replicating each concept in C ∗ u .

Original: As before, the sentence length is set as the average length of all training sentences for fair comparison.
Revised: As before, the sentence length is set as the average length of all training sentences for fair comparison.

Original: • one instance : S + contains just one instance of each concept in C ∗ u .
Revised: • one instance : S + contains just one instance of each concept in C ∗ u .

Original: Thus, the input sentence length is shorter than those of others in general.
Revised: Thus, the input sentence length is shorter than those of others in general.

Original: • inclusion : S + contains the training sentences that include at least one concept in C ∗ u
Revised: • inclusion : S + contains the training sentences that include at least one concept in C ∗ u

Original: . • random :
Revised: . • random :

Original: S + contains randomly sampled sentences from the training data
Revised: S + contains randomly sampled sentences from the training data.

Original: In contrast, µ − = 1 |S − | (cid:80) s ∈S − a u ( s ) is the average value of unit activation when forwarding S − , which consists of  sentences that do not include any concept in C ∗ u .
Revised: In contrast, µ − = 1 |S − | (cid:80) s ∈S − a u ( s ) is the average value of unit activation when forwarding S − , which consists of training sentences that do not include any concept in C ∗ u .

Original: Intuitively, if unit u ’s activation is highly sensitive to C ∗ u ( i.e . those found by our alignment method) and if it is not to other factors, then Sel u gets large; otherwise, Sel u is near 0.
Revised: Intuitively, if unit u ’s activation is highly sensitive to C ∗ u ( i.e . those found by our alignment method) and if it is not to other factors, then Sel u gets large; otherwise, Sel u is near 0.

Original: Figure 2 shows the mean and variance of selectivity values for all units learned in each dataset for the four S + categories.
Revised: Figure 2 shows the mean and variance of selectivity values for all units learned in each dataset for the four S + categories.

Original: Consistent with our intuition, in all datasets, the mean selectivity of the replicate set is the highest with a signiﬁcant margin, that of one instance, inclusion set is the runner-up, and that of the random set is the lowest.
Revised: Consistent with our intuition, in all datasets, the mean selectivity of the replicate set is the highest with a signiﬁcant margin, that of one instance, inclusion set is the runner-up, and that of the random set is the lowest.

Original: These results support our claims that units are selectively responsive to speciﬁc concepts and our method is successful to align such concepts to units.
Revised: These results support our claims that units are selectively responsive to speciﬁc concepts and our method is successful to align such concepts to units.

Original: Moreover, the mean selectivity of the replicate set is higher than that of the one instance set, which implies that a unit’s activation increases as its concepts appear more often in the input text.
Revised: Moreover, the mean selectivity of the replicate set is higher than that of the one instance set, which implies that a unit’s activation increases as its concepts appear more often in the input text.

Original: 4. C ONCEPT A LIGNMENT OF U NITS
Revised: 4. C ONCEPT A LIGNMENT OF U NITS

Original: Figure 3 shows examples of the top K sentences and the aligned concepts that are discovered by our method, for selected units.
Revised: Figure 3 shows examples of the top K sentences and the aligned concepts that are discovered by our method, for selected units.

Original: For each unit, we ﬁnd the top K = 10 sentences that activate the most in several encoding layers of ByteNet and VDCNN, and select some of them (only up to ﬁve sentences are shown due to space constraints).
Revised: For each unit, we ﬁnd the top K = 10 sentences that activate the most in several encoding layers of ByteNet and VDCNN, and select some of them (only up to ﬁve sentences are shown due to space constraints).

Original: We observe that some patterns appear frequently within the top
Revised: We observe that some patterns appear frequently within the top

Original: K sentences.
Revised: K sentences.

Original: For example, in the top K sentences that activate unit 124 of 0th layer of ByteNet, the concepts of ‘(’, ‘)’, ‘-’ appear in common, while the concepts of soft, software, wi appear frequently in the sentences for unit 19 of 1st layer of VDCNN.
Revised: For example, in the top K sentences that activate unit 124 of 0th layer of ByteNet, the concepts of ‘(’, ‘)’, ‘-’ appear in common, while the concepts of soft, software, wi appear frequently in the sentences for unit 19 of 1st layer of VDCNN.

Original: These results qualitatively show that individual units are selectively responsive to speciﬁc natural language concepts.
Revised: These results qualitatively show that individual units are selectively responsive to speciﬁc natural language concepts.

Original: More interestingly, we discover that many units could capture speciﬁc meanings or syntactic roles beyond superﬁcial, low-level patterns.
Revised: More interestingly, we discover that many units could capture speciﬁc meanings or syntactic roles beyond superﬁcial, low-level patterns.

Original: For example, unit 690 of the 14th layer in ByteNet captures ( what, who, where ) concepts, all of which play a similar grammatical role.
Revised: For example, unit 690 of the 14th layer in ByteNet captures ( what, who, where ) concepts, all of which play a similar grammatical role.

Original: On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge ( sure, know, aware ) and the VDCNN unit detecting years ( 1999, 1969, 1992 ).
Revised: On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge ( sure, know, aware ) and the VDCNN unit detecting years ( 1999, 1969, 1992 ).

Original: This suggests that, although we train character-level CNNs with feeding sentences as the form of discrete symbols ( i.e . character indices), individual units could capture natural language concepts sharing  similar semantic or grammatical role.
Revised: This suggests that, although we train character-level CNNs with feeding sentences as the form of discrete symbols ( i.e . character indices), individual units could capture natural language concepts sharing a similar semantic or grammatical role.

Original: More quantitative analyses for such concepts are available in Appendix E.
Revised: More quantitative analyses for such concepts are available in Appendix E.

Original:  We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts.
Revised: [Table] [Figure] We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts.

Original: For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top
Revised: For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top

Original: K sentences, all top K sentences have similar nuances like positive sentiments.
Revised: K sentences, all top K sentences have similar nuances like positive sentiments.

Original: In this case, our method does capture relevant phrase-level concepts (e.g., very disappointing, absolute worst place ), indicating that the higher-level nuance (e.g., negativity) is indirectly captured.
Revised: In this case, our method does capture relevant phrase-level concepts (e.g., very disappointing, absolute worst place ), indicating that the higher-level nuance (e.g., negativity) is indirectly captured.

Original: [Table] [Figure] We note that, because the number of morphemes, words, and phrases present in training corpus is usually much greater than the number of units per layer, we do not expect to always align any natural language concepts in the corpus to one of the units.
Revised: We note that, because the number of morphemes, words, and phrases present in training corpus is usually much greater than the number of units per layer, we do not expect to always align any natural language concepts in the corpus to one of the units.

Original: Our approach thus tends to ﬁnd concepts that are frequent in training data or considered as more important than others for solving the target task.
Revised: Our approach thus tends to ﬁnd concepts that are frequent in training data or considered as more important than others for solving the target task.

Original: Overall, these results suggest how input sentences are represented in the hidden layers of the CNN:
Revised: Overall, these results suggest how input sentences are represented in the hidden layers of the CNN:

Original: • Several units in the CNN learned on NLP tasks respond selectively to speciﬁc natural language concepts, rather than getting activated in an uninterpretable way.
Revised: • Several units in the CNN learned on NLP tasks respond selectively to speciﬁc natural language concepts, rather than getting activated in an uninterpretable way.

Original: This means that these units can serve as detectors for speciﬁc natural language concepts.
Revised: This means that these units can serve as detectors for speciﬁc natural language concepts.

Original:  • There are units capturing syntactically or semantically related concepts, suggesting that they model the meaning or grammatical role shared between those concepts , as opposed to superﬁcially modeling each natural language symbol .[Figure] [Figure]
Revised: [Figure] • There are units capturing syntactically or semantically related concepts, suggesting that they model the meaning or grammatical role shared between those concepts , as opposed to superﬁcially modeling each natural language symbol .

Original: 4. C ONCEPT D ISTRIBUTION IN L AYERS
Revised: 4. C ONCEPT D ISTRIBUTION IN L AYERS

Original: Using the concept alignments found earlier, we can visualize how concepts are distributed across layers.
Revised: Using the concept alignments found earlier, we can visualize how concepts are distributed across layers.

Original: Figure 4 shows the concepts of the units in the 0th, 1st, 3rd layer of VDCNN learned on AGNews dataset, and 0th, 4th, and 14th layer of the ByteNet encoder learned on English-to-German
Revised: Figure 4 shows the concepts of the units in the 0th, 1st, 3rd layer of VDCNN learned on AGNews dataset, and 0th, 4th, and 14th layer of the ByteNet encoder learned on English-to-German

Original: Europarl dataset with their number of aligned units.
Revised: Europarl dataset with their number of aligned units.

Original: For each layer, we sort concepts ina decreasing order by the number of aligned units and show 30 concepts most aligned.
Revised: For each layer, we sort concepts in decreasing order by the number of aligned units and show 30 concepts most aligned.

Original: Recall that, since we align concepts for each unit, there are concepts aligned to multiple units simultaneously.
Revised: Recall that, since we align concepts for each unit, there are concepts aligned to multiple units simultaneously.

Original: Concept distribution for other datasets are available in Appendix G.
Revised: Concept distribution for other datasets are available in Appendix G.

Original: Overall, we ﬁnd that data and task-speciﬁc concepts are likely to be aligned to many units.
Revised: Overall, we ﬁnd that data and task-speciﬁc concepts are likely to be aligned to many units.

Original: In AG
Revised: In AG

Original: News, since the task is to classify given sentences into following categories; World , Sports , Business and Science/Tech , concepts related to these topics commonly emerge.
Revised: News, since the task is to classify given sentences into following categories; World , Sports , Business and Science/Tech , concepts related to these topics commonly emerge.

Original: Similarly, we can see that units learned for Europarl dataset focus to encode some key words ( e.g . vote, propose, environment ) in the training corpus.
Revised: Similarly, we can see that units learned for Europarl dataset focus to encode some key words ( e.g . vote, propose, environment ) in the training corpus.

Original: 4. H OW DOES C ONCEPT G RANULARITY E VOLVE WITH L AYER ?
Revised: 4. H OW DOES C ONCEPT G RANULARITY E VOLVE WITH L AYER ?

Original: In computer vision tasks, visual concepts captured by units in CNN representations learned for image recognition tasks evolve with layer depths; color, texture concepts are emergent in earlier layers and more abstract concepts like parts and objects are emergent in deeper layers.
Revised: In computer vision tasks, visual concepts captured by units in CNN representations learned for image recognition tasks evolve with layer depths; color, texture concepts are emergent in earlier layers and more abstract concepts like parts and objects are emergent in deeper layers.

Original: To conﬁrm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and N -gram phrase ( N = 2 , 3 , 4 , 5 ), and observe the number of units that they are aligned in different layers.
Revised: To conﬁrm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and N -gram phrase ( N = 2 , 3 , 4 , 5 ), and observe the number of units that they are aligned in different layers.

Original: Figure 5 shows this trend, where in lower layers such as the 0th layer, less phrase concepts but more morphemes and words are detected.
Revised: Figure 5 shows this trend, where in lower layers such as the 0th layer, fewer phrase concepts but more morphemes and words are detected.

Original: This is because we use a character-level CNN, whose receptive ﬁelds of convolution may not be large enough to detect lengthy phrases.
Revised: This is because we use a character-level CNN, whose receptive ﬁelds of convolution may not be large enough to detect lengthy phrases.

Original: Further, interestingly in translation cases, we observe that concepts signiﬁcantly change in shallower layers ( e.g . from the 0th to the 4th), but do not change much from middle to deeper layers ( e.g . from the 5th to the 14th).
Revised: Further, interestingly in translation cases, we observe that concepts signiﬁcantly change in shallower layers ( e.g . from the 0th to the 4th), but do not change much from middle to deeper layers ( e.g . from the 5th to the 14th).

Original: Thus, it remains for us to answer the following question: for the representations learned on translation datasets, why does concept granularity not evolve much in deeper layers?
Revised: Thus, it remains for us to answer the following question: for the representations learned on translation datasets, why does concept granularity not evolve much in deeper layers?

Original: One possibility is that the capacity of the network is large enough so that the representations in  middle layers could be sufﬁciently informative to solve the task.
Revised: One possibility is that the capacity of the network is large enough so that the representations in the middle layers could be sufﬁciently informative to solve the task.

Original: To validate this hypothesis, we re-train ByteNet from scratch while varying only layer depth of the encoder and ﬁxing other conditions.
Revised: To validate this hypothesis, we re-train ByteNet from scratch while varying only layer depth of the encoder and ﬁxing other conditions.

Original: We record their BLEU scores on the validation data as shown in Figure 6.
Revised: We record their BLEU scores on the validation data as shown in Figure 6.

Original: The performance of the translation model does not change much with more than six encoder layers, but it signiﬁcantly drops at the models with fewer than 4 encoder layers.
Revised: The performance of the translation model does not change much with more than six encoder layers, but it signiﬁcantly drops at the models with fewer than 4 encoder layers.

Original: This trend coincides with the result from Figure 5 that the evolution of concept granularity stops around middle-to-higher layers.
Revised: This trend coincides with the result from Figure 5 that the evolution of concept granularity stops around middle-to-higher layers.

Original: This shared pattern suggests that about six encoder layers are enough to encode informative factors in the given datasets to perform optimally on the translation task.
Revised: This shared pattern suggests that about six encoder layers are enough to encode informative factors in the given datasets to perform optimally on the translation task.

Original: In deeper models, this may suggest that the middlelayer’s representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative. e n c 01 e n c 02 e n c 04 e n c 06 e n c 08 e n c# of encoder layers 12.
Revised: In deeper models, this may suggest that the middle e n c 01 e n c 02 e n c 04 e n c 06 e n c 08 e n c# of encoder layers 12.

Original: B L E U en-fr-news en-de-news en-cs-news
Revised: B L E U en-fr-news en-de-news en-cs-news

Original: [Figure]
Revised: [Figure]

Original: ) (ii)
Revised: layer’s representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

Original: 4. W HAT M AKES C ERTAIN C ONCEPTS E MERGE M ORE THAN O THERS ?
Revised: 4. W HAT M AKES C ERTAIN C ONCEPTS E MERGE M ORE THAN O THERS ?

Original: We show how many units each concept is aligned per layer in Section 4.3 and Appendix G. We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.
Revised: We show how many units each concept is aligned per layer in Section 4.4 and Appendix G. We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

Original: Then, the following question arises: What makes certain concepts emerge more than others?
Revised: Then, the following question arises: What makes certain concepts emerge more than others?

Original: Two possible hypotheses may explain the emergence of dominant concepts.
Revised: Two possible hypotheses may explain the emergence of dominant concepts.

Original: First, the concepts with  higher frequency in training data may be aligned to more units.
Revised: First, the concepts with a higher frequency in training data may be aligned to more units.

Original: Figure 7-(i) shows the correlation between the frequency of each concept in the training corpus and the number of units where each concept is aligned in the last layer of the english-to-french translation model.
Revised: Figure 7-(a) shows the correlation between the frequency of each concept in the training corpus and the number of units where each concept is aligned in the last layer of the topic classiﬁcation model learned on AG News dataset.

Original: Second, the concepts that have more inﬂuence on the objective function (expected loss) may be aligned to more units.
Revised: Second, the concepts that have more inﬂuence on the objective function (expected loss) may be aligned to more units.

Original: We can measure the effect of concept c on the task performance as Delta of
Revised: We can measure the effect of concept c on the task performance as Delta of

Original: Expected Loss (DEL) as follows:
Revised: Expected Loss (DEL) as follows:

Original: [Equation]
Revised: [Equation]

Original: where S is a set of training sentences, and Y is the set of ground-truths, and L ( s, y ) is the loss function for the input sentence s and label y .
Revised: where S is a set of training sentences, and Y is the set of ground-truths, and L ( s, y ) is the loss function for the input sentence s and label y .

Original: Occ c ( s ) is an occlusion of concept c in sentence s , where we replace concept c by dummy character tokens that have no meaning.
Revised: Occ c ( s ) is an occlusion of concept c in sentence s , where we replace concept c by dummy character tokens that have no meaning.

Original: If sentence s does not include concept c , Occ c ( s ) equals to original sentence s .
Revised: If sentence s does not include concept c , Occ c ( s ) equals to original sentence s .

Original: As a result, DEL ( c ) measures the impact of concept c on the loss function, where a large positive value implies that concept c has an important role for solving the target task.
Revised: As a result, DEL ( c ) measures the impact of concept c on the loss function, where a large positive value implies that concept c has an important role for solving the target task.

Original: Figure 7-(ii) shows the correlation between the DEL and the number of units per concept.
Revised: Figure 7-(b) shows the correlation between the DEL and the number of units per concept.

Original: The Pearson correlation coefﬁcients for the hypothesis (i) and (ii) are 0.482 / 0.531, respectively.
Revised: The Pearson correlation coefﬁcients for the hypothesis (a) and (b) are 0.732 / 0.492, respectively.

Original: Such high values implicate that the representations are learned for identifying the frequent concepts in the training data and important concepts for solving the target task.
Revised: Such high values implicate that the representations are learned for identifying the frequent concepts in the training data and important concepts for solving the target task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a simple but highly effective concept alignment method for character-level CNNs to conﬁrm that each unit of the hidden layers serves as detectors of natural language concepts.
Revised: We proposed a simple but highly effective concept alignment method for character-level CNNs to conﬁrm that each unit of the hidden layers serves as detectors of natural language concepts.

Original: Using this method, we analyzed the characteristics of units with multiple datasets on classiﬁcation and translation tasks.
Revised: Using this method, we analyzed the characteristics of units with multiple datasets on classiﬁcation and translation tasks.

Original: Consequently, we shed light on how deep representations capture the natural language, and how they vary with various conditions.
Revised: Consequently, we shed light on how deep representations capture the natural language, and how they vary with various conditions.

Original: An interesting future direction is to extend the concept coverage from natural language to more abstract forms such as sentence structure, nuance, and tone.
Revised: An interesting future direction is to extend the concept coverage from natural language to more abstract forms such as sentence structure, nuance, and tone.

Original: Another direction is to quantify the properties of individual units in other models widely used in NLP tasks.
Revised: Another direction is to quantify the properties of individual units in other models widely used in NLP tasks.

Original: In particular, combining our deﬁnition of concepts with the attention mechanism ( e.g .
Revised: In particular, combining our deﬁnition of concepts with the attention mechanism ( e.g .

Original: Bahdanau et al.
Revised: Bahdanau et al.

Original: (2015)) could be a promising direction, because it can reveal how the representations are attended by the model to capture concepts, helping us better understand the decision-making process of popular deep models. 
Revised: (2015)) could be a promising direction, because it can reveal how the representations are attended by the model to capture concepts, helping us better understand the decision-making process of popular deep models. A CKNOWLEDGMENTS

Revised: We appreciate Insu Jeon, Jaemin Cho, Sewon Min, Yunseok Jang and the anonymous reviewers for their helpful comments and discussions.

Revised: This work was supported by Kakao and Kakao Brainzt corporations, and Creative-Pioneering Researchers Program through Seoul National University.

Revised: Gunhee Kim is the corresponding author

Original: A O THER M ETRICS FOR D O A WITH B IASED A LIGNMENT R ESULT
Revised: A O THER M ETRICS FOR D O A WITH B IASED A LIGNMENT R ESULT

Original: In section 3.3, we deﬁne Degree of Alignment (DoA) between concept c n and unit u as activation value of unit u for replication of c n .
Revised: In section 3.2, we deﬁne Degree of Alignment (DoA) between concept c n and unit u as activation value of unit u for replication of c n .

Original: We tried lots of stuff while we were working on DoA metrics, but a lot of it gives biased concept alignment result for several reasons.
Revised: We tried lots of stuff while we were working on DoA metrics, but a lot of it gives biased concept alignment result for several reasons.

Original: We here provide the things we tried and their reasons for failure.
Revised: We here provide the things we tried and their reasons for failure.

Original: A. P OINT - WISE M UTUAL I NFORMATION
Revised: A. P OINT - WISE M UTUAL I NFORMATION

Original: Point-wise Mutual Information (PMI) is a measure of association used in information theory and statistics.
Revised: Point-wise Mutual Information (PMI) is a measure of association used in information theory and statistics.

Original: The PMI of a pair of samples x and y sampled from random variables X
Revised: The PMI of a pair of samples x and y sampled from random variables X

Original: and Y quantiﬁes the discrepancy between the probability of their coincidence as follows:
Revised: and Y quantiﬁes the discrepancy between the probability of their coincidence as follows:

Original: [Equation]
Revised: [Equation]

Original: We then deﬁne DoA between candidate concept c n and unit u by using PMI as follow:
Revised: We then deﬁne DoA between candidate concept c n and unit u by using PMI as follow:

Original: [Equation]
Revised: [Equation]

Original: where

Original: [Equation] 5a [Equation] However, this metric has a bias of always preferring lengthy concepts even in earlier layers, which is not possible considering the receptive ﬁeld of the convolution.
Revised: However, this metric has a bias of always preferring lengthy concepts even in earlier layers, which is not possible considering the receptive ﬁeld of the convolution.

Original: Our intuition for this bias is consistent with Role & Nadif (2011), where it is a well-known problem with PMI, which is its tendency to give very high association scores to pairs involving low-frequency ones, as the denominator is small in such cases.
Revised: Our intuition for this bias is consistent with Role & Nadif (2011), where it is a well-known problem with PMI, which is its tendency to give very high association scores to pairs involving low-frequency ones, as the denominator is small in such cases.

Original: If certain concept c n in top K sentences is very lengthy, then its frequency in the corpus p ( c n ) would get very small, and pmi( u , c n ) would be large with regardless of correlation between u and c n .
Revised: If certain concept c n in top K sentences is very lengthy, then its frequency in the corpus p ( c n ) would get very small, and pmi( u , c n ) would be large with regardless of correlation between u and c n .

Original: A. C ONCEPT O CCLUSION
Revised: A. C ONCEPT O CCLUSION

Original: We tested concept alignments with the following concept occlusion method.
Revised: We tested concept alignments with the following concept occlusion method.

Original: For each of  top K sentences, we replace a c n by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value.
Revised: For each of the top K sentences, we replace a c n by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value.

Original: We repeat this for every candidate concept in the sentences – as a result, we can identify which candidate concept greatly reduce unit activation values.
Revised: We repeat this for every candidate concept in the sentences – as a result, we can identify which candidate concept greatly reduce unit activation values.

Original: We thus deﬁne concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top K sentences.
Revised: We thus deﬁne concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top K sentences.

Original: More formally, for each unit u , let S = { s 1 , ..., s K } be top K activated sentences.
Revised: More formally, for each unit u , let S = { s 1 , ..., s K } be top K activated sentences.

Original: Since we occlude each candidate concept in sentences, we deﬁne the set of candidate concept C
Revised: Since we occlude each candidate concept in sentences, we deﬁne the set of candidate concept C

Original: = { c 1 , ..., c N } , obtained from parsing each sentence in S .
Revised: = { c 1 , ..., c N } , obtained from parsing each sentence in S .

Original: We deﬁne the degree of alignment (DoA) between a concept c ∈ C and a unit u as:
Revised: We deﬁne the degree of alignment (DoA) between a concept c ∈ C and a unit u as:

Original: [Equation]
Revised: [Equation]

Original: where Z is a normalizing factor, and a u indicates the mean activation of unit u , Occ c n ( s ) is a sentence s where candidate concept c n is occluded, and 1 ( c n ∈ s ) is an indicator of whether c n is included in the sentence s .
Revised: where Z is a normalizing factor, and a u indicates the mean activation of unit u , Occ c n ( s ) is a sentence s where candidate concept c n is occluded, and 1 ( c n ∈ s ) is an indicator of whether c n is included in the sentence s .

Original: In short, the DoA measures how much a candidate concept contributes to the activation of the unit’s top K sentences.
Revised: In short, the DoA measures how much a candidate concept contributes to the activation of the unit’s top K sentences.

Original: If a candidate concept c n appears in the top K sentences S and greatly reduces the activation of unit u , then DoA u,c n gets large, implying that the c n is strongly aligned to unit u .
Revised: If a candidate concept c n appears in the top K sentences S and greatly reduces the activation of unit u , then DoA u,c n gets large, implying that the c n is strongly aligned to unit u .

Original: Unfortunately, this metric could not fairly compare  attribution of several candidate concepts.
Revised: Unfortunately, this metric could not fairly compare the attribution of several candidate concepts.

Original: For example, consider the following two concepts c 1 = hit , c 2 = hit the ball are included in one sentence.
Revised: For example, consider the following two concepts c 1 = hit , c 2 = hit the ball are included in one sentence.

Original: Occluding c 2 might gives relatively large decrement in unit activation value than that of c 1 , since cincludes c 2 .
Revised: Occluding c 2 might gives relatively large decrement in unit activation value than that of c 1 , since c 1 includes c 2 .

Original: For this reason, the occlusion based metric is unnecessarily dependant of the length of concept, rather than it’s attribution.
Revised: For this reason, the occlusion based metric is unnecessarily dependant of the length of concept, rather than it’s attribution.

Original: A. I NCLUSION S ELECTIVITY
Revised: A. I NCLUSION S ELECTIVITY

Original: Note that inclusion selectivity in section 4.1 is also used as DoA. Recall that inclusion selectivity is calculated as equation 2.
Revised: Note that inclusion selectivity in section 4.2 is also used as DoA. Recall that inclusion selectivity is calculated as equation 2.

Original: In this case, µ + = [Equation] s ) is the average value of unit activation when forwarding a set of sentences S + , where S + denotes that sentences including candidate concept c n .
Revised: In this case, µ + = 1 |S + | (cid:80) s ∈S + a u ( s ) is the average value of unit activation when forwarding a set of sentences S + , where S + denotes that sentences including candidate concept c n .

Original: However, it induces a bias which is similar to section A.1.
Revised: However, it induces a bias which is similar to section A.1.

Original: It always prefers lengthy phrases since those lengthy concepts occur few times in entire corpus.
Revised: It always prefers lengthy phrases since those lengthy concepts occur few times in entire corpus.

Original: For example, assume that the activation value of unit u for the sentence including speciﬁc lengthy phase is very high.
Revised: For example, assume that the activation value of unit u for the sentence including speciﬁc lengthy phase is very high.

Original: If such a phrase occur only one time over the entire corpus, µ + is equal to the activation value of the sentence, which is relatively very high than µ + for other candidate concepts.
Revised: If such a phrase occurs only one time over the entire corpus, µ + is equal to the activation value of the sentence, which is relatively very high than µ + for other candidate concepts.

Original: This error could be alleviated on  very large corpus where every candidate concept occurs enough in the corpus so that estimation of µ + get relatively accurate, which is practically not possible.
Revised: This error could be alleviated on a very large corpus where every candidate concept occurs enough in the corpus so that estimation of µ + get relatively accurate, which is practically not possible.

Original: A. C OMPUTING DOA V ALUES WITHOUT R EPLICATION
Revised: A. C OMPUTING DOA V ALUES WITHOUT R EPLICATION

Original: In Section 3.3, we replicate each candidate concept into the input sentence for computing DoA in
Revised: In Section 3.2, we replicate each candidate concept into the input sentence for computing DoA in

Original: Eq.(1).
Revised: Eq.(1).

Original: Since each unit works as a concept detector whose activation value increases with the length of the input sentence (Section 4.1), it is essential to normalize the length of input for fair comparison of DoA values between the concepts that have different lengths one another.
Revised: Since each unit works as a concept detector whose activation value increases with the length of the input sentence (Section 4.2), it is essential to normalize the length of input for fair comparison of DoA values between the concepts that have different lengths one another.

Original: Without the lengthnormalization ( i.e . each input sentence consists of just one instance of the candidate concept), the
Revised: Without the lengthnormalization ( i.e . each input sentence consists of just one instance of the candidate concept), the

Original: DoA metric has a bias to prefer lengthy concepts ( e.g . phrases) because they typically have more signals that affect the unit activation than short candidate concepts ( e.g . single words).
Revised: DoA metric has a bias to prefer lengthy concepts ( e.g . phrases) because they typically have more signals that affect the unit activation than short candidate concepts ( e.g . single words).

Original: B T RAINING D ETAILS
Revised: B T RAINING D ETAILS

Original: In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classiﬁcation tasks, both to analyze properties of representations for language.
Revised: In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classiﬁcation tasks, both to analyze properties of representations for language.

Original: Training details are as follows.
Revised: Training details are as follows.

Original: [Equation]
Revised: B. B YTE N ET

Original: We trained a ByteNet on the translation tasks, in particular on the WMT’17 English-to-German
Revised: We trained a ByteNet on the translation tasks, in particular on the WMT’17 English-to-German

Original: Europarl dataset, the English-to-German news dataset, WMT’16 English-to-French, English-toCzech news dataset.
Revised: Europarl dataset, the English-to-German news dataset, WMT’16 English-to-French, English-toCzech news dataset.

Original: We used the same model architecture and hyperparameters for both datasets.
Revised: We used the same model architecture and hyperparameters for both datasets.

Original: We set the batch size to 8 and the learning rate to 0.001.
Revised: We set the batch size to 8 and the learning rate to 0.001.

Original: The parameters were optimized with
Revised: The parameters were optimized with

Original: Adam (Kingma & Ba, 2015) for 5 epochs, and early stopping was actively used for ﬁnding parameters that generalize well.
Revised: Adam (Kingma & Ba, 2015) for 5 epochs, and early stopping was actively used for ﬁnding parameters that generalize well.

Original: Our code is based on a TensorFlow (Abadi et al., 2015) implementation of
Revised: Our code is based on a TensorFlow (Abadi et al., 2015) implementation of

Original: ByteNet found in https://github.com/paarthneekhara/byteNet-tensorflow .
Revised: ByteNet found in https //github com/paarthneekhara/byteNet-tensorflow .

Original: [Equation]
Revised: B. V ERY D EEP CNN (VDCNN)

Original: We trained a VDCNN for classiﬁcation tasks, in particular on the AG News dataset, the binarized version of the Yelp Reviews dataset, and DBpedia ontology dataset.
Revised: We trained a VDCNN for classiﬁcation tasks, in particular on the AG News dataset, the binarized version of the Yelp Reviews dataset, and DBpedia ontology dataset.

Original: For each task, we used 1 temporal convolutional layer, 4 convolutional blocks with each convolutional layer having a ﬁlter width of 3.
Revised: For each task, we used 1 temporal convolutional layer, 4 convolutional blocks with each convolutional layer having a ﬁlter width of 3.

Original: In our experiments, we analyze representations of each convolutional block layer.
Revised: In our experiments, we analyze representations of each convolutional block layer.

Original: The number of units in each layer representation is 64, 128, 256, 512 respectively.
Revised: The number of units in each layer representation is 64, 128, 256, 512 respectively.

Original: We set the batch size to 64 and the learning rate to 0.01.
Revised: We set the batch size to 64 and the learning rate to 0.01.

Original: The parameters are optimized using SGD optimizer for 50 epochs, and early stopping is actively used.
Revised: The parameters are optimized using SGD optimizer for 50 epochs, and early stopping is actively used.

Original: For each of the AG News, Yelp Reviews and DBpedia datasets, a VDCNN was learned with the same structure and hyperparameters.
Revised: For each of the AG News, Yelp Reviews and DBpedia datasets, a VDCNN was learned with the same structure and hyperparameters.

Original: Our code is based on a TensorFlow implementation of VDCNN found in https://github.com/zonetrooper32/VDCNN .
Revised: Our code is based on a TensorFlow implementation of VDCNN found in https://github.com/zonetrooper32/VDCNN .

Original: C V ARIANTS OF A LIGNMENT WITH D IFFERENT M VALUES
Revised: C V ARIANTS OF A LIGNMENT WITH D IFFERENT M VALUES

Original: In Section 3.3, we set M = 3 .
Revised: In Section 3.2, we set M = 3 .

Original: Although M is used as a threshold to set how many concepts per unit are considered, different M values have little inﬂuence on quantitative results such as selectivity in Section 4.1.
Revised: Although M is used as a threshold to set how many concepts per unit are considered, different M values have little inﬂuence on quantitative results such as selectivity in Section 4.2.

Original: Figure 8 shows the mean and variance of selectivity values with different M =
Revised: Figure 8 shows the mean and variance of selectivity values with different M =

Original: [1 , 3 , 5 , 10] , where there is little variants in the overall trend; the sensitivity of the replicate set is the highest, and that of one instance is runner-up, and that of random is the lowest.
Revised: [1 , 3 , 5 , 10] , where there is little variants in the overall trend; the sensitivity of the replicate set is the highest, and that of one instance is runner-up, and that of random is the lowest.

Original: D N ON - INTERPRETABLE U NITS

Original: Whereas some units are sensitive to speciﬁc natural language concepts as shown in Section 4.2, other unites are not sensitive to any concepts at all.

Original: We call such units as non-interpretable units , which deserve to be explored.

Original: We ﬁrst deﬁne the unit interpretability for unit u as follows:

Original: [Equation]

Original: where S is the set of training sentences, a u ( s ) is the activation value of unit u , and r i is the activation value of the sentence that is made up of replicating concept

Original: c i .

Original: We deﬁne unit u as interpretable when its Interpretability ( u ) equals to 1, and otherwise as non-interpretable.

Original: The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

Original: Figure 9 shows the ratio of the interpretable units in each layer on several datasets.

Original: We observe that more than 90% of units are interpretable across all layers and all datasets.

Original: 0 1 2 3 4 5 6 7 8 9 10 11 12 13layer index 0.3 0.
Revised: 0 1 2 3 4 5 6 7 8 9 10 11 12 13layer index 0.3 0.

Original: R a t i o o f
Revised: R a t i o o f

Original: N o n - i n t e r p r e t a b l e
Revised: N o n - i n t e r p r e t a b l e

Original: U n i t s en-cs-news ag yelp en-fr-news en-de-news dbpedia en-de-europarl
Revised: U n i t s en-cs-news ag yelp en-fr-news en-de-news dbpedia en-de-europarl

Original: [Figure]
Revised: [Figure]

Original: • At some point, theparty is going to end
Revised: What would it cost the

Original: [Figure]
Revised: [Figure]

Original: • At the other extreme are

Original: Chile

Original: , Costa Rica

Original: [Figure]
Revised: [Figure]

Original: and Uruguay
Revised: So what exactly were Parliament' s requirements?

Original: [Figure]
Revised: [Figure]

Original: You

Original: gather

Original: data

Original: do experiments read , and write.
Revised: I must also thank the authors of the amendments.

Original: • Cashews, shrimp, and cocaine. • Scotland has a parliament, Wales an assembly. • What would it cost the organic producers? So what exactly were Parliament' s requirements?
Revised: [Figure]

Original: • I must also thank the authors of the amendments.

Original: • We have to recognise the difficult cases. 
Revised: We have to recognise the difficult cases. [Figure]

Original: • I shall therefore vote against the proposals.
Revised: I shall therefore vote against the proposals.

Original: [Figure]
Revised: [Figure]

Original: Figure 10 illustrates some examples of non-interpretable units with their top ﬁve activated sentences and their concepts.
Revised: Whereas some units are sensitive to speciﬁc natural language concepts as shown in Section 4.3, other unites are not sensitive to any concepts at all. We call such units as non-interpretable units , which deserve to be explored. We ﬁrst deﬁne the unit interpretability for unit u as follows:

Revised: [Equation]

Revised: where S is the set of training sentences, a u ( s ) is the activation value of unit u , and r i is the activation value of the sentence that is made up of replicating concept

Revised: c i .

Revised: We deﬁne unit u as interpretable when its Interpretability ( u ) equals to 1, and otherwise as non-interpretable.

Revised: The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

Revised: Figure 9 shows the ratio of the interpretable units in each layer on several datasets.

Revised: We observe that more than 90% of units are interpretable across all layers and all datasets.

Revised: Figure 10 illustrates some examples of non-interpretable units with their top ﬁve activated sentences and their concepts.

Original: Unlike Figure 3, the aligned concepts do not appear frequently over top-activated sentences.
Revised: Unlike Figure 3, the aligned concepts do not appear frequently over top-activated sentences.

Original: This result is obvious given that the concepts have little inﬂuence on unit activation.
Revised: This result is obvious given that the concepts have little inﬂuence on unit activation.

Original: There are several reasons why non-interpretable units appear.
Revised: There are several reasons why non-interpretable units appear.

Original: One possibility is that several units align concepts that are out of natural language form.
Revised: One possibility is that several units align concepts that are out of natural language form.

Original: For example, in unit 001 in the left of Figure 10, we discover that sentence structure involves many commas in top activated sentences.
Revised: For example, in unit 001 in the left of Figure 10, we discover that sentence structure involves many commas in top activated sentences.

Original: Since we limit the candidate concepts to only the form of morpheme, word and phrase, such punctuation concepts are hard to be detected.
Revised: Since we limit the candidate concepts to only the form of morpheme, word and phrase, such punctuation concepts are hard to be detected.

Original: Another possibility is that some units may be so-called dead units that are not sensitive to any concept at all.
Revised: Another possibility is that some units may be so-called dead units that are not sensitive to any concept at all.

Original: For example,the unit 260 in the right of Figure 10 has no pattern that appears consistently in top activated sentences.
Revised: For example, unit 260 in the right of Figure 10 has no pattern that appears consistently in top activated sentences.

Original: E C ONCEPT CLUSTERS
Revised: E C ONCEPT CLUSTERS

Original: We introduce some units whose concepts have the shared meaning in Section 4.2.
Revised: We introduce some units whose concepts have the shared meaning in Section 4.3.

Original: We here refer concept cluster to the concepts that are aligned to the same unit and have similar semantics or grammatical roles.
Revised: We here refer concept cluster to the concepts that are aligned to the same unit and have similar semantics or grammatical roles.

Original: We analyze how clusters are formed in the units and how they vary with the target task and layer depth.
Revised: We analyze how clusters are formed in the units and how they vary with the target task and layer depth.

Original: E. C ONCEPT C LUSTERS BY T ARGET T ASKS
Revised: E. C ONCEPT C LUSTERS BY T ARGET T ASKS

Original: Figure 11 illustrates some concept clusters of units in the ﬁnal layer learned on each task.
Revised: Figure 11 illustrates some concept clusters of units in the ﬁnal layer learned on each task.

Original: Top and left dendrograms of each ﬁgure show hierarchical clustering results of 30 concepts aligned with the largest number of units.
Revised: Top and left dendrograms of each ﬁgure show hierarchical clustering results of 30 concepts aligned with the largest number of units.

Original: We use clustering algorithm of M¨ullner (2011); we deﬁne the distance between two concepts as the Euclidean distance of their vector space embedding.
Revised: We use clustering algorithm of M ¨ ullner (2011); we deﬁne the distance between two concepts as the Euclidean distance of their vector space embedding.

Original: We use fastText (Bojanowski et al., 2017) pretrained on Wikipedia dataset to project each concept into the vector space.
Revised: We use fastText (Bojanowski et al., 2017) pretrained on Wikipedia dataset to project each concept into the vector space.

Original: Since fastText is a character-level N -gram based word embedding, we can universally obtain the embedding for morphemes as well as words or phrases.
Revised: Since fastText is a character-level N -gram based word embedding, we can universally obtain the embedding for morphemes as well as words or phrases.

Original: For phrase embedding, we split it to words, project each of them and average their embeddings.
Revised: For phrase embedding, we split it to words, project each of them and average their embeddings.

Original: The distance between two clusters is deﬁned as the distance between their centroids.
Revised: The distance between two clusters is deﬁned as the distance between their centroids.

Original: Each central heat map represents the number of times each concept pair is aligned to the same unit.
Revised: Each central heat map represents the number of times each concept pair is aligned to the same unit.

Original: Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings.
Revised: Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings.

Original: In Figure 11, the units learned in the classiﬁcation tasks tend to have stronger concept clusters than those learned in the translation tasks.
Revised: In Figure 11, the units learned in the classiﬁcation tasks tend to have stronger concept clusters than those learned in the translation tasks.

Original: Particularly, the concept clusters are highly evident in the units learned in DBpedia and AG News dataset.
Revised: Particularly, the concept clusters are highly evident in the units learned in DBpedia and AG News dataset.

Original: Our intuition is that units might have more beneﬁts to solve the task by clustering similar concepts in the classiﬁcation than the translation.
Revised: Our intuition is that units might have more beneﬁts to solve the task by clustering similar concepts in the classiﬁcation than the translation.

Original: That is, in the classiﬁcation, input sentences that have the similar concepts tend to belong to the same class label, while in the translation, different concepts should be translated to different words or phrases even if they have similar meanings in general. 
Revised: That is, in the classiﬁcation, input sentences that have the similar concepts tend to belong to the same class label, while in the translation, different concepts should be translated to different words or phrases even if they have similar meanings in general. [Figure] g e [Figure]

Revised: a i r

Original: E. C ONCEPT C LUSTERS BY L AYERS
Revised: [Figure] s i [Figure] i t

Revised: [Figure]

Revised: c

Revised: [Figure]

Revised: c

Revised: [Figure]

Revised: p

Revised: [Figure]

Revised: s

Revised: [Figure]

Revised: a v e r a g e p a i r w i s e s i m il a r i t y

Revised: o f a li g n e d c o n c e p

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: E. C ONCEPT C LUSTERS BY L AYERS

Original: We analyze how concept clusters change by layer in each task.
Revised: We analyze how concept clusters change by layer in each task.

Original: We compute the averaged pairwise distance between the concepts in each layer.
Revised: We compute the averaged pairwise distance between the concepts in each layer.

Original: We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014),
Revised: We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014),

Original: (2) ConceptNet (Speer et al., 2017), (3) fastText.
Revised: (2) ConceptNet (Speer et al., 2017), (3) fastText.

Original: Glove and fastText embeddings are pretrained on Wikipedia dataset, and ConcpetNet is pretrained based on the ConceptNet graph structure.
Revised: Glove and fastText embeddings are pretrained on Wikipedia dataset, and ConcpetNet is pretrained based on the ConceptNet graph structure.

Original: Figure 12 shows the averaged pairwise distances in each layer.
Revised: Figure 12 shows the averaged pairwise distances in each layer.

Original: In all tasks, there is a tendency that the concepts in the same unit become closer in the vector space as the layer goes deeper.
Revised: In all tasks, there is a tendency that the concepts in the same unit become closer in the vector space as the layer goes deeper.

Original: It indicates that individual units in earlier layers tend to capture more basic text patterns or symbols, while units in deeper layers capture more abstract semantics.
Revised: It indicates that individual units in earlier layers tend to capture more basic text patterns or symbols, while units in deeper layers capture more abstract semantics.

Original: [Figure]
Revised: [Figure]

Original: F W HAT M AKES C ERTAIN C ONCEPTS E MERGE M ORE THAN O THERS ?: O THER D ATASETS
Revised: F W HAT M AKES C ERTAIN C ONCEPTS E MERGE M ORE THAN O THERS ?: O THER D ATASETS

Original: We investigate why certain concepts emerge more than others at Section 4.5 when the ByteNet is trained on English-to-French news dataset.
Revised: We investigate why certain concepts emerge more than others at Section 4.6 when the ByteNet is trained on English-to-French news dataset.

Original: Here, Figure 14 shows more results in other datasets.
Revised: Here, Figure 14 shows more results in other datasets.

Original: Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept.
Revised: Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept.

Original: It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task.
Revised: It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task.

Original: G C ONCEPT D ISTRIBUTION IN L AYERS FOR O THER D ATASETS
Revised: G C ONCEPT D ISTRIBUTION IN L AYERS FOR O THER D ATASETS

Original: In section 4.3, we visualized how concepts are distributed across layers, where the model is trained on AG News dataset and English-to-German Europarl dataset.
Revised: In section 4.4, we visualized how concepts are distributed across layers, where the model is trained on AG News dataset and English-to-German Europarl dataset.

Original: Here, Figure 15 shows concept distribution in other datasets noted in Table 1.
Revised: Here, Figure 15 shows concept distribution in other datasets noted in Table 1.

Original: In the classiﬁcation tasks, we expect to ﬁnd more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation.
Revised: In the classiﬁcation tasks, we expect to ﬁnd more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation.

Original: While our goal is not to relate each[Figure] concept to one of the labels, we ﬁnd several concepts that are more predictive to a particular label than others.
Revised: While our goal is not to relate each concept to one of the labels, we ﬁnd several concepts that are more predictive to a particular label than others.

Original: Consistent with section 4.3, there are data-speciﬁc and task-speciﬁc concepts aligned in each layer; i.e . { worst, 2 stars, awful } at Yelp Review, { ﬁlm, ship, school } at DBpedia, and some key words at translation datasets.
Revised: Consistent with section 4.4, there are data-speciﬁc and task-speciﬁc concepts aligned in each layer; i.e . { worst, 2 stars, awful } at Yelp Review, { ﬁlm, ship, school } at DBpedia, and some key words at translation datasets.

Original: Note that Yelp Review and DBpedia is a classiﬁcation dataset, where the model is required to predict the polarity ( i.e . +1 or -1) or ontology ( i.e . Company, Educational
Revised: Note that Yelp Review and DBpedia is a classiﬁcation dataset, where the model is required to predict the polarity ( i.e . +1 or -1) or ontology ( i.e . Company, Educational

Original: Institution, Artist, Athlete, Ofﬁceholder, Mean of Transportation, Building, Natural Place, Village,
Revised: Institution, Artist, Athlete, Ofﬁceholder, Mean of Transportation, Building, Natural Place, Village,

Original: Animal, Plant, Album, Film, Written Work) for given sentence in supervised setting.
Revised: Animal, Plant, Album, Film, Written Work) for given sentence in supervised setting.

Original: H M ULTIPLE O CCURRENCES OF E ACH C ONCEPT AT D IFFERENT L AYERS
Revised: H M ULTIPLE O CCURRENCES OF E ACH C ONCEPT AT D IFFERENT L AYERS

Original: Figure 16 shows the number of occurrences of each concept at different layers.
Revised: Figure 16 shows the number of occurrences of each concept at different layers.

Original: We count how many times each concept appears across all layers and sort them ina decreasing order.
Revised: We count how many times each concept appears across all layers and sort them in decreasing order.

Original: We select two concepts in the translation model and seven concepts in the classiﬁcation model, as to their number of occurrences.
Revised: We select two concepts in the translation model and seven concepts in the classiﬁcation model, as to their number of occurrences.

Original: For example, since there are 15 encoder layers in the ByteNet translation model, we select 30 concepts in total.
Revised: For example, since there are 15 encoder layers in the ByteNet translation model, we select 30 concepts in total.

Original: Although task and data speciﬁc concepts emerge at different layers, there is no strong pattern between the concepts and their occurrences at multiple layers.
Revised: Although task and data speciﬁc concepts emerge at different layers, there is no strong pattern between the concepts and their occurrences at multiple layers.

Original: ; t a x a ll f a r a p r o b l e m g l o b a l w h
Revised: ; t a x a ll f a r a p r o b l e m g l o b a l w h

Original: o s e
Revised: o s e

Original: r
Revised: r

Original: i s k
Revised: i s k

Original: d
Revised: d

Original: e
Revised: e

Original: f
Revised: f

Original: i
Revised: i

Original: c
Revised: c

Original: i t [ # ] A r a b n e v e r [ # ] g e s t s o c
Revised: i t [ # ] A r a b n e v e r [ # ] g e s t s o c

Original: i a l d e m o c
Revised: i a l d e m o c

Original: r a c y [ # ] F i n a n c t o [ # ] s o m e e c o n o m
Revised: r a c y [ # ] F i n a n c t o [ # ] s o m e e c o n o m

Original: i c w a r [ # ] C O [ # ] r
Revised: i c w a r [ # ] C O [ # ] r

Original: i s k e c o n o m i
Revised: i s k e c o n o m i

Original: c r e c o v e r y t
Revised: c r e c o v e r y t

Original: h
Revised: h

Original: i s n e [Table] r e a li t y t
Revised: i s n e w r e a li t y t

Original: h e s e t w o s c
Revised: h e s e t w o s c

Original: e n a r i o p
Revised: e n a r i o p

Original: h
Revised: h

Original: y s i
Revised: y s i

Original: [Table]
Revised: o

Original: l o gi c
Revised: l o g

Original: [Table]
Revised: i c a l s t a b l e r e g i o n a l m o r e e ff e c t

Original: r e g i o n a l m o r e e ff e c t i v e
Revised: i v e m u c h g r e a ta r o l e

Original: [Table]
Revised: [Table]

Original: m

Original: [Table]
Revised: [Table]

Original: e r n i
Revised: n t

Original: [Table]
Revised: [Table]

Original: a t i
Revised: m o

Original: [Table] n t[Table]
Revised: d e r n i z a t i o n t

Original: s
Revised: h

Original: [Table]

Original:  [ # ] P r e s i d e n t m o n t hd e m o c
Revised: o s e [ # ] P r e s i d e n t m o n t h

Original:  r a t i c
Revised: d e m o c r a t i c

Original: i s l a m t
Revised: i s l a m t

Original: o w a r w i t
Revised: o w a r w i t

Original: h r u ss i a a m illi o n ?
Revised: h r u ss i a a m illi o n ?

Original: [ # ] N A " s o c
Revised: [ # ] N A " s o c

Original: i a li s m " h a v e o n l y 3 .s
Revised: i a li s m " h a v e o n l y 3 .s

Original: h o u l d b e d
Revised: h o u l d b e d

Original: o
Revised: o

Original: [ # ] p a r t i c
Revised: [ # ] p a r t i c

Original: u l a r w o u l d
Revised: u l a r w o u l d

Original: [ # ] M i n i r i s k m ili t a r y 1945 [ # ] [Table] a l a $ [Table]
Revised: [ # ] M i n i r i s k m ili t a r y 1945 [ # ] c a l a $m illi o n l o ss 183 , 9 p o p e l a s t a ll hu

Original: m illi
Revised: g e c o s t

Original: [Table]
Revised: [Table]

Original: n
Revised: i

Original: [Table]
Revised: [Table]

Original: [Table] p [Table] p e l a s
Revised: k li k e l y m ili t

Original: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: s
Revised: y

Original: [Table]
Revised: [Table]

Original: k li
Revised: h

Original: [Table]
Revised: [Table]

Original: l y
Revised: s

Original: [Table]
Revised: [Table]

Original: c a n a n d s h
Revised: c a n a n d s h

Original: o u l d [ # ] b o rr o
Revised: o u l d [ # ] b o rr o

Original: w p r
Revised: w p r

Original: a g m a t i c f o u r r e s e r v e r e f o r m [ # ] d e v e l o p [ # ] L a b [ # ] !
Revised: a g m a t i c f o u r r e s e r v e r e f o r m [ # ] d e v e l o p [ # ] L a b [ # ] !

Original: [ # ] C h i m u s t
Revised: [ # ] C h i m u s t

Original: d
Revised: d

Original: o t h e
Revised: o t h e

Original: s e t e c hn o
Revised: s e t e c hn o

Original: l o g yr
Revised: l o g yr

Original: e f o r m i n g h
Revised: e f o r m i n g h

Original: i
Revised: i

Original: d e [ # ] b a rr i s h
Revised: d e [ # ] b a rr i s h

Original: o u l d b e p r o m o t e i na l o n e [ # ] N s i x b illi o n s o w h a t ?
Revised: o u l d b e p r o m o t e i na l o n e [ # ] N s i x b illi o n s o w h a t ?

Original: [ # ] W e s t
Revised: [ # ] W e s t

Original: [ # ] s y s t e m o n ( 4 )t
Revised: [ # ] s y s t e m o n ( 4 )t

Original: h e s e
Revised: h e s e

Original: r e f o r m en-cs-news n e w f o u r r
Revised: r e f o r m en-cs-news n e w f o u r r

Original: i s k
Revised: i s k

Original: [ # ] s p e c
Revised: [ # ] s p e c

Original: i
Revised: i

Original: a l w h
Revised: a l w h

Original: o s e n e t a g r ee m e n t i n r e c e n t y e a r s o rr y
Revised: o s e n e t a g r ee m e n t i n r e c e n t y e a r s o rr y

Original: [ # ] a i r t
Revised: [ # ] a i r t

Original: o u r
Revised: o u r

Original: i s m e c o n o m i c a n d s o c
Revised: i s m e c o n o m i c a n d s o c

Original: i a l r i t u a l [ # ]a w e d n e s d a y
Revised: i a l r i t u a l [ # ]a w e d n e s d a y

Original: [ # ] Q u [ # ] l e g a l p r o b l e m s o l v e n ee d i n i t i a t i v e o n a m e n d m e n t t h
Revised: [ # ] Q u [ # ] l e g a l p r o b l e m s o l v e n ee d i n i t i a t i v e o n a m e n d m e n t t h

Original: i s o pp o r t un i t
Revised: i s o pp o r t un i t

Original: y r e f o r m t
Revised: y r e f o r m t

Original: h e r i g h t d e c
Revised: h e r i g h t d e c

Original: i s i o n t
Revised: i s i o n t

Original: h e p r
Revised: h e p r

Original: e
Revised: e

Original: s i d e n t [ # ] c o n c e r n s e c o n d l y
Revised: s i d e n t [ # ] c o n c e r n s e c o n d l y

Original: [ # ][ # ] c e r t a i n t h e s e c o n den-de-europarl g oo g l e a pp l e c o m p u t e r i n c .
Revised: [ # ][ # ] c e r t a i n t h e s e c o n den-de-europarl g oo g l e a pp l e c o m p u t e r i n c .

Original: c o r p o r a t
Revised: c o r p o r a t

Original: e
Revised: e

Original: e s p n 2 p r o b a s k e t
Revised: e s p n 2 p r o b a s k e t

Original: b a ll t
Revised: b a ll t

Original: h e b o s t o n r e d s o x o l y m p
Revised: h e b o s t o n r e d s o x o l y m p

Original: i c t e a m c
Revised: i c t e a m c

Original: o a c h
Revised: o a c h

Original: o l
Revised: o l

Original: y m p
Revised: y m p

Original: i c t h e s t
Revised: i c t h e s t

Original: o c k m a r k e t p e
Revised: o c k m a r k e t p e

Original: o p l e s o f t i n c .
Revised: o p l e s o f t i n c .

Original: m a n
Revised: m a n

Original: c h e s t e r , e n g l a n d s a t u r d a y n i g
Revised: c h e s t e r , e n g l a n d s a t u r d a y n i g

Original: h t
Revised: h t

Original: i s l a m i c m ili t
Revised: i s l a m i c m ili t

Original: a n t s o il f u t u r e s p a l e s t i n i a n l e a d e r s
Revised: a n t s o il f u t u r e s p a l e s t i n i a n l e a d e r s

Original: h e r o l y m p
Revised: h e r o l y m p

Original: i c ji n x b o m b a r d i e r i n c .
Revised: i c ji n x b o m b a r d i e r i n c .

Original: o f s p
Revised: o f s p

Original: a c e t e c hn
Revised: a c e t e c hn

Original: o l o g y f
Revised: o l o g y f

Original: r o m f o r d m o t
Revised: r o m f o r d m o t

Original: o r c o .
Revised: o r c o .

Original: t o g e t h e r \\ g oo g l e c a p t a i n [ # ]
Revised: t o g e t h e r \\ g oo g l e c a p t a i n [ # ]

Original: c o o f t
Revised: c o o f t

Original: h e i r
Revised: h e i r

Original: h
Revised: h

Original: o s t a g e sag
Revised: o s t a g e sag

Original: i s a m a z i n g [ # ] ?
Revised: i s a m a z i n g [ # ] ?

Original: o
Revised: o

Original: w o w o w o w o w o w n o b e s t !
Revised: w o w o w o w o w o w n o b e s t !

Original: y u m !
Revised: y u m !

Original: h a t e i t w a s m y f
Revised: h a t e i t w a s m y f

Original: i r s t t i m e n e v e r a n d e v e r n e v e r k w a s f a n t a s t i c
Revised: i r s t t i m e n e v e r a n d e v e r n e v e r k w a s f a n t a s t i c

Original: l o v e t h
Revised: l o v e t h

Original: i s p
Revised: i s p

Original: i zz a r e a l l y g oo d i ' v e h a d b e tt e r ....
Revised: i zz a r e a l l y g oo d i ' v e h a d b e tt e r ....

Original: [ # ] c k b u dd y l o v e , l o v e , l o v e .
Revised: [ # ] c k b u dd y l o v e , l o v e , l o v e .

Original: m e d i o c r e m il k s h a k e s h e r e !!!!
Revised: m e d i o c r e m il k s h a k e s h e r e !!!!

Original: \ n \ n l e a s i n gyelp [ # ] i t y
Revised: \ n \ n l e a s i n gyelp [ # ] i t y

Original: [ # ] o [ # ] u s b y
Revised: [ # ] o [ # ] u s b y

Original: [ # ] cc [ # ] p l a c e s c h oo l f il m s
Revised: [ # ] cc [ # ] p l a c e s c h oo l f il m s

Original: c o m p a n y [ # ] 199 l o c a t e d e d u c
Revised: c o m p a n y [ # ] 199 l o c a t e d e d u c

Original: a t
Revised: a t

Original: i o n a l c o ll e g e
Revised: i o n a l c o ll e g e

Original: [ # ] s u
Revised: [ # ] s u

Original: p e r f o un d e d
Revised: p e r f o un d e d

Original: [ # ] a m a b a n k [ # ] c o b r o w n [ # ] w r
Revised: [ # ] a m a b a n k [ # ] c o b r o w n [ # ] w r

Original: i t e r [ # ] v ill e k e nn e t h
Revised: i t e r [ # ] v ill e k e nn e t h

Original: [ # ] i n s u r j a n u a r y 25t
Revised: [ # ] i n s u r j a n u a r y 25t

Original: h e r o y a l n a vy p
Revised: h e r o y a l n a vy p

Original: o li t i c i a n .
Revised: o li t i c i a n .

Original: b o r n
Revised: b o r n

Original: [Figure]
Revised: [Figure]