Original: D ISCOVERING L ATENT C ONCEPTS L EARNED IN BERT
Revised: D ISCOVERING L ATENT C ONCEPTS L EARNED IN BERT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A large number of studies analyze deep neural network models and their ability to encode various linguistic and non-linguistic concepts, and subsequently provide an interpretation of the inner mechanics of these models.
Revised: A large number of studies analyze deep neural network models and their ability to encode various linguistic and non-linguistic concepts, and subsequently provide an interpretation of the inner mechanics of these models.

Original: A major caveat to these studies, however, is that their scope of interpretation is limited to pre-deﬁned concepts that mainly reinforce the traditional linguistic knowledge.
Revised: A major caveat to these studies, however, is that their scope of interpretation is limited to pre-deﬁned concepts that mainly reinforce the traditional linguistic knowledge.

Original: Consequently, the interpretation does not reﬂect on novel concepts learned by a model.
Revised: Consequently, the interpretation does not reﬂect on novel concepts learned by a model.

Original: We address this limitation by discovering and analyzing latent concepts learned in a model in an unsupervised fashion, thus providing interpretation from the model’s perspective.
Revised: We address this limitation by discovering and analyzing latent concepts learned in a model in an unsupervised fashion, thus providing interpretation from the model’s perspective.

Original: In this work, we study i) what latent concepts exist in the pre-trained BERT model, ii) how the discovered latent concepts align or diverge from the classical linguistic hierarchy and iii) how the concepts evolve across layers.
Revised: In this work, we study i) what latent concepts exist in the pre-trained BERT model, ii) how the discovered latent concepts align or diverge from the classical linguistic hierarchy and iii) how the concepts evolve across layers.

Original: Some notable ﬁndings of our analysis include: i) the model learns novel concepts such as animal categories and demographic groups, which do not strictly adhere to any predeﬁned categorization (e.g., POS, semantic tags), ii) several latent concepts are based on multiple properties which may include semantics, syntax or morphology, iii) lower layers dominate in learning shallow lexical concepts while higher layers learn semantic relations and iv) the discovered latent concepts highlight potential biases learned in the model.
Revised: Some notable ﬁndings of our analysis include: i) the model learns novel concepts such as animal categories and demographic groups, which do not strictly adhere to any predeﬁned categorization (e.g., POS, semantic tags), ii) several latent concepts are based on multiple properties which may include semantics, syntax or morphology, iii) lower layers dominate in learning shallow lexical concepts while higher layers learn semantic relations and iv) the discovered latent concepts highlight potential biases learned in the model.

Original: We also release a novel BERT concept dataset ( BCD ) consisting of 198 concept labels and 1M annotated instances.
Revised: We also release a novel BERT concept dataset ( BCD ) consisting of 183 concept labels and 1M annotated instances.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Interpreting deep neural networks (DNNs) is essential for understanding their inner working and for their successful deployment to real world scenarios, especially for applications where fairness, trust, accountability, reliability and ethical decision-making are considered critically important metrics.
Revised: Interpreting deep neural networks (DNNs) is essential for understanding their inner working and for their successful deployment to real world scenarios, especially for applications where fairness, trust, accountability, reliability and ethical decision-making are considered critically important metrics.

Original: A large number of studies have been devoted towards interpreting DNNs.
Revised: A large number of studies have been devoted towards interpreting DNNs.

Original: A major line of work in interpreting deep NLP models reﬂects on their ability to learn various language concepts (Tenney et al., 2019b; Liu et al., 2019).
Revised: A major line of work in interpreting deep NLP models reﬂects on their ability to learn various language concepts (Tenney et al., 2019b; Liu et al., 2019a).

Original: More speciﬁcally, they rely on pre-deﬁned linguistic concepts such as: parts-of-speech tags and semantic tags, and probe whether the speciﬁc linguistic knowledge is learned in various parts of the network.
Revised: More speciﬁcally, they rely on pre-deﬁned linguistic concepts such as: parts-of-speech tags and semantic tags, and probe whether the speciﬁc linguistic knowledge is learned in various parts of the network.

Original: For example, Belinkov et al.
Revised: For example, Belinkov et al.

Original: (2020) found that lower layers in the NMT model capture word morphology and the higher layers learn long-range dependencies.
Revised: (2020) found that lower layers in the NMT model capture word morphology and the higher layers learn long-range dependencies.

Original: A pitfall to this line of research is its sole reliance on pre-deﬁned concepts which may only provide a narrow view of what the model knows, largely ignoring any latent concepts learned within these models.
Revised: A pitfall to this line of research is its sole reliance on pre-deﬁned concepts which may only provide a narrow view of what the model knows, largely ignoring any latent concepts learned within these models.

Original: Another drawback of using user-deﬁned concepts is the involvement of human bias in choosing a concept which may result in a misleading interpretation.
Revised: Another drawback of using user-deﬁned concepts is the involvement of human bias in choosing a concept which may result in a misleading interpretation.

Original: In this work, we target this issue and approach interpretability from a model’s perspective by discovering latent concepts in pre-trained models.
Revised: In this work, we target this issue and approach interpretability from a model’s perspective by discovering latent concepts in pre-trained models.

Original: Mikolov et al. (2013); Reif et al.
Revised: Mikolov et al. (2013); Reif et al.

Original: (2019) showed that words group together in the high dimensional space based on syntactic and semantic relationships.
Revised: (2019) showed that words group together in the high dimensional space based on syntactic and semantic relationships.

Original: We hypothesize that these groupings represent latent concepts , a model learns about the language.
Revised: We hypothesize that these groupings represent latent concepts , a model learns about the language.

Original: To this end, we cluster contextualized representations in high dimensional space and study the relation behind each group.
Revised: To this end, we cluster contextualized representations in high dimensional space and study the relation behind each group.

Original: More speciﬁcally, we use hierarchical clustering to identify groups of related words.
Revised: More speciﬁcally, we use hierarchical clustering to identify groups of related words.

Original: We manually annotate the groups and assemble a concept dataset.
Revised: We manually annotate the groups and assemble a concept dataset.

Original: This is the ﬁrst concept dataset that enables model-centric interpretation and will serve as a benchmark for analyzing such models.
Revised: This is the ﬁrst concept dataset that enables model-centric interpretation and will serve as a benchmark for analyzing such models.

Original: We base our study on BERT.
Revised: We base our study on BERT (Devlin et al., 2019b).

Original: However, our method can be generically applied to other models.
Revised: However, our method can be generically applied to other models.

Original: We study i) what latent concepts exist in BERT, ii) how the discovered latent concepts align with pre-deﬁned concepts, and iii) how the concepts evolve across layers.
Revised: We study i) what latent concepts exist in BERT, ii) how the discovered latent concepts align with pre-deﬁned concepts, and iii) how the concepts evolve across layers.

Original: Our analysis reveals interesting ﬁndings such as: i) the model learns novel concepts like the hierarchy in animal kingdom, demographic hierarchy etc.
Revised: Our analysis reveals interesting ﬁndings such as: i) the model learns novel concepts like the hierarchy in animal kingdom, demographic hierarchy etc.

Original: which do not strictly adhere to any pre-deﬁned concepts, ii) the lower layers focus on shallow lexical concepts while the higher layers learn semantic relations, iii) the concepts learned at higher layers are more aligned with the linguistically motivated concepts compared to the lower layers, and iv) the discovered latent concepts highlight potential biases learned in the model.
Revised: which do not strictly adhere to any pre-deﬁned concepts, ii) the lower layers focus on shallow lexical concepts while the higher layers learn semantic relations, iii) the concepts learned at higher layers are more aligned with the linguistically motivated concepts compared to the lower layers, and iv) the discovered latent concepts highlight potential biases learned in the model.

Original: There are two main contributions of our work: i) we interpret BERT by analyzing latent concepts learned in the network, ii) we provide a ﬁrst multi-facet hierarchical concept dataset ( BCD ) that addresses a major limitation of a prominent class of interpretation studies.
Revised: There are two main contributions of our work: i) we interpret BERT by analyzing latent concepts learned in the network, ii) we provide a ﬁrst multi-facet hierarchical concept dataset ( BCD ) that addresses a major limitation of a prominent class of interpretation studies.

Original: Our proposed concept pool facilitates interpretation from the perspective of the model and will serve as a common benchmark for future research.
Revised: Our proposed concept pool facilitates interpretation from the perspective of the model and will serve as a common benchmark for future research.

Original: Moreover, it enriches existing linguistic and pre-deﬁned concepts and has the potential to serve as a new classiﬁcation dataset for NLP in general.
Revised: Moreover, it enriches existing linguistic and pre-deﬁned concepts and has the potential to serve as a new classiﬁcation dataset for NLP in general.

Original: BCD consists of 198 ﬁne-grained concept labels with a total of 1M annotated instances.
Revised: BCD consists of 183 ﬁne-grained concept labels with a total of 1M annotated instances.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The interpretation of DNNs in NLP has been largely dominated by the post-hoc model analysis (Liu et al., 2019;
Revised: The interpretation of DNNs in NLP has been largely dominated by the post-hoc model analysis (Liu et al., 2019a;

Original: Shi et al., 2016; Belinkov et al., 2020; Tenney et al., 2019a) where the primary question is what linguistic concepts are encoded in the representation (Radford et al., 2018; Lakretz et al., 2019; Dalvi et al., 2019; Durrani et al., 2020).
Revised: Shi et al., 2016; Belinkov et al., 2020; Tenney et al., 2019a) where the primary question is what linguistic concepts are encoded in the representation (Radford et al., 2018; Lakretz et al., 2019; Dalvi et al., 2019; Durrani et al., 2020).

Original: The studies have analyzed a range of concepts varying from low-level shallow concepts such as word position and sentence length to linguistically motivated concepts such as parts of speech (POS), syntax and semantic tags.
Revised: The studies have analyzed a range of concepts varying from low-level shallow concepts such as word position and sentence length to linguistically motivated concepts such as parts of speech (POS), syntax and semantic tags.

Original: They provide interesting insights into what concepts are learned and encoded in the representation.
Revised: They provide interesting insights into what concepts are learned and encoded in the representation.

Original: One of the shortcomings of these studies is their reliance on pre-deﬁned concepts for interpretation.
Revised: One of the shortcomings of these studies is their reliance on pre-deﬁned concepts for interpretation.

Original: Consequently, novel concepts that models learn about the language are largely undiscovered.
Revised: Consequently, novel concepts that models learn about the language are largely undiscovered.

Original: Recently Michael et al.
Revised: Recently Michael et al.

Original: (2020) analyzed latent concepts learned in pre-trained models.
Revised: (2020) analyzed latent concepts learned in pre-trained models.

Original: They used a binary classiﬁcation task to induce latent concepts relevant to that task and showed the presence of linguistically motivated and novel concepts in the representation.
Revised: They used a binary classiﬁcation task to induce latent concepts relevant to that task and showed the presence of linguistically motivated and novel concepts in the representation.

Revised: Mamou et al.

Revised: (2020) analyzed representations of pre-trained models using mean-ﬁeld theoretic manifold analysis and showed the emergence of linguistic structure across layer depth.

Original: Similar to Michael et al.
Revised: Similar to Michael et al.

Revised: ; Mamou et al.

Original: (2020), we aim to analyze latent concepts learned in the representation.
Revised: (2020), we aim to analyze latent concepts learned in the representation.

Original: However different from them, we intend to analyze raw representation independent of a post-hoc classiﬁcation task.
Revised: However different from them, we intend to analyze raw representations independent of a supervised task.

Original: The reliance on a supervision task effects the type of latent concepts found and may not fully reﬂect the latent concepts encoded in the raw representation.
Revised: The reliance on a supervision task effects the type of latent concepts found and may not fully reﬂect the latent concepts encoded in the raw representation.

Original: Moreover, the post-hoc classiﬁers require a careful evaluation to mitigate the concerns regarding the power of the probe (Hewitt & Liang, 2019; Zhang & Bowman, 2018).
Revised: Moreover, the post-hoc classiﬁers require a careful evaluation to mitigate the concerns regarding the power of the probe (Hewitt & Liang, 2019; Zhang & Bowman, 2018).

Original: In this work, we address these limitations by analyzing latent concepts learned in BERT in an unsupervised fashion.
Revised: In this work, we address these limitations by analyzing latent concepts learned in BERT in an unsupervised fashion.

Original: In addition, we provide a novel multi-facet hierarchical concept dataset that signiﬁcantly enrich already existing pre-deﬁned concepts.
Revised: In addition, we provide a novel multi-facet hierarchical concept dataset that signiﬁcantly enrich already existing pre-deﬁned concepts.

Original:  W HAT IS A CONCEPT ?
Revised:  W HAT IS A CONCEPT ?

Original: A concept represents a notion and can be viewed as a coherent fragment of knowledge.
Revised: A concept represents a notion and can be viewed as a coherent fragment of knowledge.

Original: Stock (2010) deﬁnes concept as “a class containing certain objects as elements, where the objects have certain properties”.
Revised: Stock (2010) deﬁnes concept as “a class containing certain objects as elements, where the objects have certain properties”.

Original: Deveaud et al.
Revised: Deveaud et al.

Original: (2014) considers latent concepts as words that convey the most information or that are the most relevant to the initial query.
Revised: (2014) considers latent concepts as words that convey the most information or that are the most relevant to the initial query.

Original: A concept in language can be anything ranging from a shallow lexical property such as character bigram, to a complex syntactic phenomenon such as a adjectival phrase or a semantic property such as an event.
Revised: A concept in language can be anything ranging from a shallow lexical property such as character bigram, to a complex syntactic phenomenon such as a adjectival phrase or a semantic property such as an event.

Original: Here, we loosely deﬁne concept as a group of words that are meaningful where meaningful can be a lexical, semantic, syntactic, morphological and/or any other relationship.
Revised: Here, we loosely deﬁne concept as a group of words that are meaningful where meaningful can be a lexical, semantic, syntactic, morphological and/or any other relationship.

Original: For example, names of ice-hockey teams form a semantic cluster, words that occur in a certain position of a sentence represent a syntactic concept, words that are ﬁrst person singular verbs form a morphological cluster, words that begin with ”anti” form a lexical cluster.
Revised: For example, names of ice-hockey teams form a semantic cluster, words that occur in a certain position of a sentence represent a syntactic concept, words that are ﬁrst person singular verbs form a morphological cluster, words that begin with ”anti” form a lexical cluster.

Original: In our implementation, we consider clusters of word contextual representations in
Revised: In our implementation, we consider clusters of word contextual representations in

Original: BERT as hidden concept candidates and the human-annotated candidates with linguistic meanings as our discovered hidden concepts.
Revised: BERT as hidden concept candidates and the human-annotated candidates with linguistic meanings as our discovered hidden concepts.

Original: However, if a human is unable to identify a relation behind a group of words i.e. not-meaningful, we consider it as uninterpretable .
Revised: However, if a human is unable to identify a relation behind a group of words i.e. not-meaningful, we consider it as uninterpretable .

Original: Since, a relation may exist that humans are unable to comprehend.
Revised: Since, a relation may exist that humans are unable to comprehend.

Original: Input : −→ y l = word representation of all running words

Original: Parameter : K

Original: = No. of output clus- ters 1: for each word

Original: w i do 2: assign w i to cluster c i 3: end for 4: while No. of clusters (cid:54) =

Original: K do 5: for each cluster pair

Original: c i , c i (cid:48) do 6: d i,i (cid:48) =

Original: inner-cluster difference of combined cluster c i and c i (cid:48) 7: end for 8: c j , c j (cid:48)

Original: =

Original: cluster pair with minimum value of d 9: merge clusters c j and c j (cid:48) 10: end while

Original: Input : X train = word representations for train data, Y train =

Original: cluster id from Algorithm 1, X test = word representations for test data,

Original: Parameter : t = probability thresh- old 1: c = unique cluster ids from Y train 2: M = train Logistic Regression model on

Original: X train and Y train 3: for each x ∈ X test do 4: p

Original: = predict K probabilities for each cluster using M and input x 5:

Original: i = arg max p 6: if p i > = t then 7: assign x to cluster id c i 8: end if 9: end for

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Consider a Neural Network (NN) model M with L layers { l 1 , l 2 , ...
Revised: Consider a Neural Network (NN) model M with L layers { l 1 , l 2 , ...

Original: l l , ..., l L } , where each layer contains H hidden nodes.
Revised: l l , ..., l L } , where each layer contains H hidden nodes.

Original: An input sentence consisting of M words w 1 , w 2 , ...w i , ..., w M is fed into NN.
Revised: An input sentence consisting of M words w 1 , w 2 , ...w i , ..., w M is fed into NN.

Original: For the i -th word input, we can compute the node output (after applying the activation functions)
Revised: For the i -th word input, we can compute the node output (after applying the activation functions)

Original: y lh ( i ) of every hidden node h ∈ { 1 , ..., H } in each layer l , where −→ y l ( i ) is the vector representation of all hidden node outputs in layer l for w i .
Revised: y lh ( i ) of every hidden node h ∈ { 1 , ..., H } in each layer l , where −→ y l ( i ) is the vector representation of all hidden node outputs in layer l for w i .

Original: Our goal is to cluster −→ y l , the contextual representation, among all i -th input words.
Revised: Our goal is to cluster −→ y l , the contextual representation, among all i -th input words.

Original: We call this clustering asour latent concepts that maps words into a group with meanings.
Revised: We call this clustering as latent concepts that map words into meaningful groups.

Original: We introduce an annotation schema and manually assign labels to each cluster following our deﬁnition of a concept.
Revised: We introduce an annotation schema and manually assign labels to each cluster following our deﬁnition of a concept.

Original: In the following sections, we discuss each step in detail.
Revised: In the following, we discuss each step in detail.

Original: 4. C LUSTERING
Revised: 4. C LUSTERING

Original: Algorithm 1  presents the clustering procedure.
Revised: Algorithm 1 (Appendix A) presents the clustering procedure.

Original: For each layer, we cluster the hidden nodes into K groups using agglomerative hierarchical clustering (Gowda & Krishna, 1978) trained on the contextualized representation.
Revised: For each layer, we cluster the hidden nodes into K groups using agglomerative hierarchical clustering (Gowda & Krishna, 1978) trained on the contextualized representation.

Original: We apply Ward’s minimum variance criterion that minimizes the total within-cluster variance.
Revised: We apply Ward’s minimum variance criterion that minimizes the total within-cluster variance.

Original: The distance between two vector representations is calculated with the squared Euclidean distance.
Revised: The distance between two vector representations is calculated with the squared Euclidean distance.

Original: The number of clusters K is a hyperparameter.
Revised: The number of clusters K is a hyperparameter.

Original: We found an optimal value of K using the Elbow method.
Revised: We found an optimal value of K using the Elbow method.

Original: Figure 2 shows the Elbow scores over a range of cluster values.
Revised: Figure 2 shows the Elbow scores over a range of cluster values.

Original: We set K = 1000 since after that point, the inertia starts decreasing linearly.
Revised: We set K = 1000 since after that point, the inertia starts decreasing linearly.

Original: 4. D ATA P REPARATION
Revised: 4. D ATA P REPARATION

Original: The choice of a dataset is an important factor to be considered when generating latent concepts.
Revised: The choice of a dataset is an important factor to be considered when generating latent concepts.

Original: Ideally, one would like to form latent clusters over a large collection with diverse domains.
Revised: Ideally, one would like to form latent clusters over a large collection with diverse domains.

Original: However, clustering a large number of high-dimensional vectors is computationally and memory intensive.
Revised: However, clustering a large number of points is computationally and memory intensive.

Original: For instance, 200k vectors each of size 768 require around 400GB of CPU memory. A potential workaround is  to reduce the dimension of word contextual vector representations before clustering.
Revised: In the case of agglomerative hierarchical clustering using Ward linkage, the memory complexity is O ( n 2 ) , which translates to about 400GB of runtime memory for 200k input vectors.

Original: However, this may cause a loss of information and the resulting clusters may not be an accurate representation of  thelatent space of BERT.
Revised: A potential workaround is to use a different linkage algorithm like single or average linkage, but each of these come with their own caveats regarding assumptions around how the space is structured and how noisy the inputs are.

Original: We address these problems by using a subset of a large dataset of News 2018 WMT 1 (359M tokens).
Revised: We address these problems by using a subset of a large dataset of News 2018 WMT 1 (359M tokens).

Original: We randomly select 250k sentences from the dataset ( ≈ 5 million tokens).
Revised: We randomly select 250k sentences from the dataset ( ≈ 5 million tokens).

Original: We further limit the number of tokens to ≈ 210k by discarding singletons, closed-class words, and tokens with a frequency higher than 10.
Revised: We further limit the number of tokens to ≈ 210k by discarding singletons, closed-class words, and tokens with a frequency higher than 10.

Revised: For every token, we extract its contextualized representations and embedding layer representation using the 12-layered BERT-base-cased model (Devlin et al., 2019a) in the following way: First, we tokenize sentences using the Moses tokenizer 2 and pass them through the standard pipeline of BERT as implemented in HuggingFace 3 .

Original: For every token, we extracted its contextualized representations and em-
Revised: We extract layer-wise representations of every token.

Original: Our new concepts represent novel aspects that are not found in thepre-deﬁned concepts. bedding layer representation using BERT (Devlin et al., 2019).
Revised: If the BERT tokenizer splits an input token into subwords or multiple tokens, we mean pool over them to create an embedding of the input token.

Original: We cluster words based on their contextualized representations.
Revised: For every layer, we cluster tokens using their contextualized representations.

Original: 4. C ONCEPT L ABELS
Revised: 4. C ONCEPT L ABELS

Original: We deﬁne a hierarchical concept-tagset, starting with the core language properties i.e., semantics, parts-of-speech, lexical, syntax, etc.
Revised: We deﬁne a hierarchical concept-tagset, starting with the core language properties i.e., semantics, parts-of-speech, lexical, syntax, etc.

Original: For each cluster, we assign the core properties that are represented by that cluster, and further enrich the label with ﬁner hierarchical information.
Revised: For each cluster, we assign the core properties that are represented by that cluster, and further enrich the label with ﬁner hierarchical information.

Original: It is worth noting that the core language properties are not mutually exclusive i.e., a cluster can belong to more than one core properties at the same time.
Revised: It is worth noting that the core language properties are not mutually exclusive i.e., a cluster can belong to more than one core properties at the same time.

Original: Consider a group of words mentioning ﬁrst name of the football players of the German team and all of the names occur at the start of a sentence , the following series of tags will be assigned to the cluster: semantic:origin:Germany , semantic:entertainment:sport:football , semantic:namedentity:person:firstname , syntax:position:firstword .
Revised: Consider a group of words mentioning ﬁrst name of the football players of the German team and all of the names occur at the start of a sentence , the following series of tags will be assigned to the cluster: [Equation] namedentity [Equation]

Original:  Here, we preserve the hierarchy at various levels such as, sport, person name, origin, etc., which can be used to combine clusters to analyze a largergroup.
Revised: firstname , [Equation] firstword . Here, we preserve the hierarchy at various levels such as, sport, person name, origin, etc., which can be used to combine clusters to analyze a larger

Original: 4. A NNOTATION T ASK
Revised: 4. A NNOTATION T ASK

Original: Following our deﬁnition of concepts, we formulated the annotation task using the two questions below.
Revised: Following our deﬁnition of concepts, we formulated the annotation task using the two questions below.

Original: The annotation task consists of looking at the cluster and answer the following questions.
Revised: The annotation task consists of looking at the cluster and answer the following questions.

Original: Q1 :
Revised: Q1 :

Original: Is the cluster meaningful?
Revised: Is the cluster meaningful?

Original: Q2 :
Revised: Q2 :

Original: Can the two clusters be combined to form a meaningful group?
Revised: Can the two clusters be combined to form a meaningful group?

Original: The answers of both questions are (i) yes, (ii) no, and (iii) don’t know or can’t judge.
Revised: The answers of both questions are (i) yes, (ii) no, and (iii) don’t know or can’t judge.

Original: Q1 mainly involves identifying if the cluster is meaningful, according to our deﬁnition of a concept (Section 3).
Revised: Q1 mainly involves identifying if the cluster is meaningful, according to our deﬁnition of a concept (Section 3).

Original: Based on the yes answer of Q1, annotators’ task is to either introduce a new concept label where it belongs or classify it into an appropriate existing concept label.
Revised: Based on the yes answer of Q1, annotators’ task is to either introduce a new concept label where it belongs or classify it into an appropriate existing concept label.

Original: Note that at the initial phase of the annotation, the existing concept label set was empty.
Revised: Note that at the initial phase of the annotation, the existing concept label set was empty.

Original: As the annotators annotate and ﬁnalize the concept labels, we accumulate them into the existing concept label set and provide them in the next round of annotations, in order to facilitate and speed-up the annotations.
Revised: As the annotators annotate and ﬁnalize the concept labels, we accumulate them into the existing concept label set and provide them in the next round of annotations, in order to facilitate and speed-up the annotations.

Original: In Q2 , we ask the annotators to identify whether two sibling clusters can be also combined to form a meaningful super-cluster.
Revised: In Q2 , we ask the annotators to identify whether two sibling clusters can be also combined to form a meaningful super-cluster.

Original: Our motivation here is to keep the hierarchical information that BERT is capturing intact.
Revised: Our motivation here is to keep the hierarchical information that BERT is capturing intact.

Original: For example, a cluster capturing Rock music bands in the
Revised: For example, a cluster capturing Rock music bands in the

Original: US can be combined with its sibling that groups Rock music bands in the UK to form a Rock music bands cluster.
Revised: US can be combined with its sibling that groups Rock music bands in the UK to form a Rock music bands cluster.

Original: Similarly, based on the yes answer of Q2, annotators’ need to assign a concept label for the super-cluster.
Revised: Similarly, based on the yes answer of Q2, annotators’ need to assign a concept label for the super-cluster.

Original: We show annotators, the contextual information (i.e., corresponding sentences) of the words for each of these questions.
Revised: We show annotators, the contextual information (i.e., corresponding sentences) of the words for each of these questions.

Original: Having contextual information enables very ﬁne-grained concepts: for example, named-entities:person can be classiﬁed into ﬁner named-entities:person:Hollywood .
Revised: Having contextual information enables very ﬁne-grained concepts: for example, named-entities:person can be classiﬁed into ﬁner named-entities:person:Hollywood .

Original: The annotation guidelines combined with the examples and screenshots of the annotation platform is provided in Appendix A.
Revised: The annotation guidelines combined with the examples and screenshots of the annotation platform is provided in Appendix B.

Original: The annotators are asked to maintain hierarchy while annotating e.g., a cluster of ice hockey will be annotated as semantic:
Revised: The annotators are asked to maintain hierarchy while annotating e.g., a cluster of ice hockey will be annotated as semantic:

Original: entertainment:sport:ice hockey since the words are semantically grouped and the group belongs to sports, which is a subcategory of entertainment.
Revised: entertainment:sport:ice hockey since the words are semantically grouped and the group belongs to sports, which is a subcategory of entertainment.

Original: In case of more than one possible properties that form a cluster, we annotate it with both properties e.g., title case and country names where the former is a lexical property and the latter is a semantic property.
Revised: In case of more than one possible properties that form a cluster, we annotate it with both properties e.g., title case and country names where the former is a lexical property and the latter is a semantic property.

Original: Since annotation is an expensive process, we carried out annotation of the ﬁnal layer of BERT only.
Revised: Since annotation is an expensive process, we carried out annotation of the ﬁnal layer of BERT only.

Original: The concept annotation is performed by three annotators followed by a consolidation step to resolve disagreements if there are any.
Revised: The concept annotation is performed by three annotators followed by a consolidation step to resolve disagreements if there are any.

Original: For the annotation task, we randomly selected 234 clusters out of 1000 clusters obtained during the clustering step (see Section 4.1).
Revised: For the annotation task, we randomly selected 234 clusters out of 1000 clusters obtained during the clustering step (see Section 4.1).

Original:  A NNOTATED D ATASET 5. I NTER A NNOTATION A GREEMENT
Revised:  A NNOTATED D ATASET 5. I NTER A NNOTATION A GREEMENT

Original: We computed Fleiss κ (Fleiss et al., 2013) and found the average value for Q1 and Q2 to be 0.61 and 0.64, respectively, which isclassiﬁed as substantial in the Kappa measurement (Landis & Koch, 1977).
Revised: We computed Fleiss κ (Fleiss et al., 2013) and found the average value for Q1 and Q2 to be 0.61 and 0.64, respectively, which is substantial in the κ measurement (Landis & Koch, 1977).

Original: More detail on κ measures and agreement computation using other metrics are reported indetail in Appendix (Table 2). 
Revised: The detail on κ measures and agreement computation using other metrics are reported in Appendix (Table 2). [Figure]

Original: 5. S TATISTICS
Revised: 5. S TATISTICS

Original: For Q1, we found 198 (84.6%) meaningful clusters and 36 (15.4%) non-meaningful clusters.
Revised: For Q1, we found 198 (84.6%) meaningful clusters and 36 (15.4%) non-meaningful clusters.

Original: For Q2, we found 142 (75.9%) clusters out of 187 clusters that form a meaningful cluster when combined with their siblings.
Revised: For Q2, we found 142 (75.9%) clusters out of 187 clusters that form a meaningful cluster when combined with their siblings.

Original: The annotation process resulted in 183 unique concept labels from theclusters (See Appendix A.3 for the complete list of labels).
Revised: The annotation process resulted in 183 unique concept labels from theclusters (See Appendix B.3 for the complete list of labels).

Original: The high number of concept labels is due to the ﬁne-grained distinctions and multi-facet nature of the clusters where a cluster can get more than one labels.
Revised: The high number of concept labels is due to the ﬁne-grained distinctions and multi-facet nature of the clusters where a cluster can get more than one labels.

Original: The label set consists of 8 lexical labels, 10 morphological labels,semantic labels and one syntactic label.
Revised: The label set consists of 20 lexical labels, 10 morphological labels, 152 semantic labels and one syntactic label.

Original: The semantic labels form the richest hierarchy where the next level of hierarchy consists of 42 labels such as: entertainment, sports, geo-politics, etc.
Revised: The semantic labels form the richest hierarchy where the next level of hierarchy consists of 42 labels such as: entertainment, sports, geo-politics etc.

Original: 5. E XAMPLES OF C ONCEPT H IERARCHY
Revised: 5. E XAMPLES OF C ONCEPT H IERARCHY

Original: In this Section, we describe the concept hierarchy that we have achieved through our annotation process.
Revised: In this Section, we describe the concept hierarchy that we have achieved through our annotation process.

Original: Figure 1 illustrates an example of a resulting tree found by hierarchical clustering.
Revised: Figure 1 illustrates an example of a resulting tree found by hierarchical clustering.

Original: Each node is a concept candidate.
Revised: Each node is a concept candidate.

Original: We capture different levels of ﬁne-grained meanings of word groups, which do not exist in pre-deﬁned concepts.
Revised: We capture different levels of ﬁne-grained meanings of word groups, which do not exist in pre-deﬁned concepts.

Original: For example, a SEM or an NE tagger would mark a name as person, but our discovery of latent concepts show that
Revised: For example, a SEM or an NE tagger would mark a name as person, but our discovery of latent concepts show that

Original: BERT uses a ﬁner hierarchy by further classifying them as “baseball players”, “soccer players”, and “athletes”.
Revised: BERT uses a ﬁner hierarchy by further classifying them as “baseball players”, “soccer players”, and “athletes”.

Original: Our annotation process preserves this information and will result in hierarchical labels such as semantic:named-entity:person:celebrity:actor and semantic:named-entity:person:athletes:soccer .
Revised: Our annotation process preserves this information and will result in hierarchical labels such as semantic:named-entity:person:celebrity:actor and semantic:named-entity:person:athletes:soccer .

Original: Due to the multi-faceted nature of the latent clusters, many of them are annotated with more than one concept label.
Revised: Due to the multi-faceted nature of the latent clusters, many of them are annotated with more than one concept label.

Original: In the case of the clusters shown in Figure 1, they have a common lexical property (title case) and hence will also get the label LEX:case:title during annotation.
Revised: In the case of the clusters shown in Figure 1, they have a common lexical property (title case) and hence will also get the label LEX:case:title during annotation.

Original: Figure 3 shows word clouds of a few annotated clusters along with their descriptions.
Revised: Figure 3 shows word clouds of a few annotated clusters along with their descriptions.

Original:  A NALYSIS AND F INDINGS 6. Q UALITATIVE A NALYSIS
Revised:  A NALYSIS AND F INDINGS 6. Q UALITATIVE A NALYSIS

Original: Our annotated dataset unveils insightful analysis of the latent concepts learned within BERT representations.
Revised: Our annotated dataset unveils insightful analysis of the latent concepts learned within BERT representations.

Original: Figure 4 presents a few examples of the discovered concepts.
Revised: Figure 4 presents a few examples of the discovered concepts.

Original: The two clusters of decimal numbers (Figure 4a, 4b) look quite similar, but are semantically different.
Revised: The two clusters of decimal numbers (Figure 4a, 4b) look quite similar, but are semantically different.

Original: The former represents numbers appearing as percentages e.g., 9.6% or 2.4 percent and the latter captures monetary ﬁgures e.g., 9.6 billion Euros or 2.4 million dollars .
Revised: The former represents numbers appearing as percentages e.g., 9.6% or 2.4 percent and the latter captures monetary ﬁgures e.g., 9.6 billion Euros or 2.4 million dollars .

Original: We found these two clusters to be sibling, which shows that BERT is learning a morpho-semantic hierarchy where it grouped all[Figure] the decimal numbers together (morphologically) and then further made a semantic distinction by dividing them into percentages and monetary ﬁgures.
Revised: We found these two clusters to be sibling, which shows that BERT is learning a morpho-semantic hierarchy where it grouped all the decimal numbers together (morphologically) and then further made a semantic distinction by dividing them into percentages and monetary ﬁgures.

Original: Such a subtle difference in the usage of decimal numbers shows the importance of using ﬁne-grained concepts for interpretation.
Revised: Such a subtle difference in the usage of decimal numbers shows the importance of using ﬁne-grained concepts for interpretation.

Original: For example, numbers are generally analyzed as one concept ( CD tag in POS) which may be less informative or even misleading given that BERT treats them in a variety of different ways.
Revised: For example, numbers are generally analyzed as one concept ( CD tag in POS) which may be less informative or even misleading given that BERT treats them in a variety of different ways.

Original: Figure 4c shows an example of a cluster where the model captures an era of time, speciﬁcally 18 and 19 hundreds.
Revised: Figure 4c shows an example of a cluster where the model captures an era of time, speciﬁcally 18 and 19 hundreds.

Original: Learning such  information may help the model to learn relation between a particular era and the events occurring in that time period.
Revised: Learning such [Figure] information may help the model to learn relation between a particular era and the events occurring in that time period.

Original: Similarly, we found BERT learning semantic distinction of animal kingdom and separated animals into land, sea and ﬂying groups (see Appendix A.3).
Revised: Similarly, we found BERT learning semantic distinction of animal kingdom and separated animals into land, sea and ﬂying groups (see Appendix B.3).

Original: These informative categories support the use of pre-trained models as a knowledge base (Petroni et al., 2019).
Revised: These informative categories support the use of pre-trained models as a knowledge base (Petroni et al., 2019).

Original: We further observed that BERT learns aspects speciﬁc to theprovided training data.
Revised: We further observed that BERT learns aspects speciﬁc to the training data.

Original: Figure 4d shows a cluster of words where female roles (Moms, Mothers, Granny, Aunt) are grouped together with job roles such as (housekeeper, Maid, Nanny).
Revised: Figure 4d shows a cluster of words where female roles (Moms, Mothers, Granny, Aunt) are grouped together with job roles such as (housekeeper, Maid, Nanny).

Original: The contextual information in the sentences where these words appear, do not explicate whether the person is a male or a female, but based on the data used for the training of BERT, it has associated these roles to females.
Revised: The contextual information in the sentences where these words appear, do not explicate whether the person is a male or a female, but based on the data used for the training of BERT, it has associated these roles to females.

Original: For example, the sentence “ Welfare staff ’s ’ good housekeeping ’ praised for uncovering disability beneﬁts error ” is a general sentence.
Revised: For example, the sentence “ Welfare staff ’s ’ good housekeeping ’ praised for uncovering disability beneﬁts error ” is a general sentence.

Original: However, the contextualized representation of housekeeping in this sentence associates it to a female-related concept.
Revised: However, the contextualized representation of housekeeping in this sentence associates it to a female-related concept.

Original: Similarly, in another example BERT clustered names based on demography even though the context in which the word is used, does not explicitly provide such information.
Revised: Similarly, in another example BERT clustered names based on demography even though the context in which the word is used, does not explicitly provide such information.

Original: An example is the sentence, “ Bookings : Dzagoev , Schennikov , Musa , Akinfeev ” where the name Musa is used in a diverse context but it is associated to a cluster of speciﬁc demography (Arab region).
Revised: An example is the sentence, “ Bookings : Dzagoev , Schennikov , Musa , Akinfeev ” where the name Musa is used in a diverse context but it is associated to a cluster of speciﬁc demography (Arab region).

Original: While it is natural to form clusters based on speciﬁc data aspects, we argue that the use of such concepts while making a general prediction raises concerns of bias and fairness.
Revised: While it is natural to form clusters based on speciﬁc data aspects, we argue that the use of such concepts while making a general prediction raises concerns of bias and fairness.

Original: For example, a loan prediction application should not rely on the background of the name of the applicant.
Revised: For example, a loan prediction application should not rely on the background of the name of the applicant.

Original: On the contrary, a clothing recommendation system may use such information to customize suggestions towards a culture or a religion.

Original: 6. A LIGNMENT WITH P RE - DEFINED C ONCEPTS
Revised: 6. A LIGNMENT WITH P RE - DEFINED C ONCEPTS

Original: A number of works on interpreting DNNs study to what extent abstract linguistic concepts such as parts-of-speech tags, semantic tags, etc are captured within the learned representations.
Revised: A number of works on interpreting DNNs study to what extent abstract linguistic concepts such as parts-of-speech tags, semantic tags, etc are captured within the learned representations.

Original: This is in view of one school of thought which believes that an NLP model needs to learn linguistic concepts in order to perform well and to generalize (Marasovi´c, 2018).
Revised: This is in view of one school of thought which believes that an NLP model needs to learn linguistic concepts in order to perform well and to generalize (Marasovi´c, 2018).

Original: For example, the holy grail in machine translation is that a proﬁcient model ought to be aware of word morphology, grammatical structure, and semantics to do well (Vauquois, 1968; Jones et al., 2012).
Revised: For example, the holy grail in machine translation is that a proﬁcient model ought to be aware of word morphology, grammatical structure, and semantics to do well (Vauquois, 1968; Jones et al., 2012).

Original: In this section, we explored how well the latent concepts in BERT align with the pre-deﬁned linguistic concepts.
Revised: In this section, we explored how well the latent concepts in BERT align with the pre-deﬁned linguistic concepts.

Original: We compared the resulting clusters with the following pre-deﬁned concepts: parts of speech (POS) tags (Marcus et al., 1993), semantic tags (SEM) using the Parallel Meaning Bank data (Abzianidze et al., 2017), CCG supertagging using CCGBank (Hockenmaier, 2006), syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz, 2000), WordNet (Miller, 1995) and LIWC psycholinguistic-based tags (Pennebaker et al., 2001).
Revised: We compared the resulting clusters with the following pre-deﬁned concepts: parts of speech (POS) tags (Marcus et al., 1993), semantic tags (SEM) using the Parallel Meaning Bank data (Abzianidze et al., 2017), CCG supertagging using CCGBank (Hockenmaier, 2006), syntactic chunking (Chunking) using CoNLL 2000 shared task dataset (Tjong Kim Sang & Buchholz, 2000), WordNet (Miller, 1995) and LIWC psycholinguistic-based tags (Pennebaker et al., 2001).

Original: We also introduced lexical and syntactic concepts such as casing, ngram matches, afﬁxes, ﬁrst and last word in a sentence etc.
Revised: We also introduced lexical and syntactic concepts such as casing, ngram matches, afﬁxes, ﬁrst and last word in a sentence etc.

Original: Appendix B provides the complete list of pre-deﬁned concepts.
Revised: Appendix D provides the complete list of pre-deﬁned concepts.

Original: For POS, SEM, Chunking, and
Revised: For POS, SEM, Chunking, and

Original: [Figure] CCG, we trained a BERT-based classiﬁer using the training data of each task and tagged the words in the selected News dataset.
Revised: CCG, we trained a BERT-based classiﬁer using the training data of each task and tagged the words in the selected News dataset.

Original: For WordNet and LIWC, we directly used their lexicon.
Revised: For WordNet and LIWC, we directly used their lexicon.

Original: We consider a latent concept to align with a corresponding pre-deﬁned concept if 90% of the cluster’s tokens have been annotated with that pre-deﬁned concept.
Revised: We consider a latent concept to align with a corresponding pre-deﬁned concept if 90% of the cluster’s tokens have been annotated with that pre-deﬁned concept.

Original: For example, Figure 4a and 4b are assigned a tag POS:CD (parts-of-speech:cardinal numbers).
Revised: For example, Figure 4a and 4b are assigned a tag POS:CD (parts-of-speech:cardinal numbers).

Original: Table 1 shows the statistics on the number of clusters matched with the pre-deﬁned concepts.
Revised: Table 1 shows the statistics on the number of clusters matched with the pre-deﬁned concepts.

Original: Of the 1000 clusters, we found the linguistic concepts in POS to have maximum alignment (30%) with the latent concepts in BERT, followed by
Revised: Of the 1000 clusters, we found the linguistic concepts in POS to have maximum alignment (30%) with the latent concepts in BERT, followed by

Original: Casing (23%).
Revised: Casing (23%).

Original: The remaining pre-deﬁned concepts including various linguistic categories aligned with fewer than 10% of the latent concepts.
Revised: The remaining pre-deﬁned concepts including various linguistic categories aligned with fewer than 10% of the latent concepts.

Original: We further explored the parallelism between the hierarchy of latent and pre-deﬁned linguistic concepts using POS tags.
Revised: We further explored the parallelism between the hierarchy of latent and pre-deﬁned linguistic concepts using POS tags.

Original: We merged the POS tags into 9 coarse tags and aligned them with the latent concepts.
Revised: We merged the POS tags into 9 coarse tags and aligned them with the latent concepts.

Original: The number of latent concepts aligned to these coarse concepts increased to 50%, showing that the model has learned coarser POS concepts and does not follow the same ﬁne-grained hierarchy.
Revised: The number of latent concepts aligned to these coarse concepts increased to 50%, showing that the model has learned coarser POS concepts and does not follow the same ﬁne-grained hierarchy.

Original: 6. L AYER - WISE C OMPARISON OF CONCEPTS
Revised: 6. L AYER - WISE C OMPARISON OF CONCEPTS

Original: How do the latent concepts evolve in the network?
Revised: How do the latent concepts evolve in the network?

Original: To answer this question, we need to manually annotate per-layer latent concepts and do a cross-layer comparison.
Revised: To answer this question, we need to manually annotate per-layer latent concepts and do a cross-layer comparison.

Original: However, as manual annotation is an expensive process, we used pre-deﬁned concepts as a proxy to annotate the concepts and compare the layers.
Revised: However, as manual annotation is an expensive process, we used pre-deﬁned concepts as a proxy to annotate the concepts and compare across layers.

Original: Figure 5 shows various interesting patterns related to lexical (LEX), syntactic (SYN), Morphological (Morph) and Semantic (SEM) concepts.
Revised: Figure 5 shows various interesting patterns related to lexical (LEX), syntactic (SYN), morphological (Morph) and semantic (SEM) concepts.

Original: Interestingly, the layers differ signiﬁcantly in terms of the concepts they capture.
Revised: Interestingly, the layers differ signiﬁcantly in terms of the concepts they capture.

Original: The lower layers tend to learn shallow linguistic concepts.
Revised: The lower layers tend to learn shallow linguistic concepts.

Original: For example, in Figure 5a – the embedding layer is dominated by the concepts based on common ngram afﬁxes.
Revised: For example, in Figure 5a – the embedding layer is dominated by the concepts based on common ngram afﬁxes.

Original: The number of ngram-based shallow lexical clusters consistently decreases in the subsequent layers.
Revised: The number of ngram-based shallow lexical clusters consistently decreases in the subsequent layers.

Original: The sufﬁxes concept showed a different trend, where the information peaks at the lower-middle layers and then drops in the higher layers.
Revised: The sufﬁxes concept showed a different trend, where the information peaks at the lower-middle layers and then drops in the higher layers.

Revised: The dominance of ngrams-based concepts in the lower layers is due to the subword segmentation.

Original: Their noticeable presence in the middle layers might be due to the fact that sufﬁxes often maintain grammatical information unlike other ngrams.
Revised: The noticeable presence in the middle layers might be due to the fact that sufﬁxes often maintain grammatical information unlike other ngrams.

Original: For example, the ngram “ome” does not have a linguistic connotation, but sufﬁx “ful” converts a noun into an adjective.
Revised: For example, the ngram “ome” does not have a linguistic connotation, but sufﬁx “ful” converts a noun into an adjective.

Original: Interestingly, casing information is least represented in the lower layers and is consistently improved for higher layers.
Revised: Interestingly, casing information is least represented in the lower layers and is consistently improved for higher layers.

Original: Casing in English has a morphological role, marking a named entity, which is an essential knowledge to preserve at the higher layers.
Revised: Casing in English has a morphological role, marking a named entity, which is an essential knowledge to preserve at the higher layers.

Original: [Table] The patterns using the classical NLP tasks: POS, SEM, CCG and Chunking all showed similar trends (See Figure 5b and 5c).
Revised: The patterns using the classical NLP tasks: POS, SEM, CCG and Chunking all showed similar trends (See Figure 5b and 5c).

Original: The concepts were predominantly captured at the middle and higher layers, with minimal representation at the lower layers.
Revised: The concepts were predominantly captured at the middle and higher layers, with minimal representation at the lower layers.

Original: WordNet and LIWC, which are cognitive and psychology based grouping of words respectively, showed a rather different trend.
Revised: WordNet and LIWC, which are cognitive and psychology based grouping of words respectively, showed a rather different trend.

Original: These concepts had a higher match at the embedding layer and a consistent drop after the middle layers.
Revised: These concepts had a higher match at the embedding layer and a consistent drop after the middle layers.

Original: In other words, the higher and last layers of BERT are less aligned with the cognitive and psychology based grouping of words, and these concepts are better represented at the embedding and lower layers.
Revised: In other words, the higher and last layers of BERT are less aligned with the cognitive and psychology based grouping of words, and these concepts are better represented at the embedding and lower layers.

Revised: We manually investigate a few clusters from the lower layers that match with WordNet and LIWC.

Revised: We found that these clusters group words based on similar meaning e.g. all words related to ”family relations” or ”directions in space” form a cluster (See Appendix C for more examples).

Original: Lastly, Figure 5c also shows that the FirstWord position information is predominantly learned at the last layers only.
Revised: Lastly, Figure 5c also shows that the FirstWord position information is predominantly learned at the last layers only.

Original: This is somewhat unexpected, given that no clusters were based on this concept in the embedding layer, despite it having an explicitly position component.
Revised: This is somewhat unexpected, given that no clusters were based on this concept in the embedding layer, despite it having an explicitly position component.

Original: The overall trend of lower-layers learning shallow concepts (ngram, afﬁxes) and middle to higher layers learning linguistic concepts (POS, SEM) andsyntactic concepts (FirstWord, Chunking, CCG) aligns with the ﬁndings of Geva et al. 
Revised: Based on these trends, we can speculate about the evolution of knowledge across the network: lower layers encode the lexical and meaning-related knowledge and with the inclusion of context in the higher layers, the encoded concepts evolve into representing linguistic hierarchy. These ﬁndings are in line with Mamou et al. (2020) where they found the emergence of linguistic manifolds across layer depth. 6. U NALIGNED C LUSTERS

Original: (2020), who also found that lower layers of the feed-forward network in thetransformer blocks capture shallow lexical concepts and the higher layers to be more focused on the semantic concepts. 
Revised: In this section, we discuss the concepts that were not matched with the pre-deﬁned concepts. We have divided them into 3 categories: i) latent concepts that can be explained via composition of Lexical [Table]

Original: BERT C ONCEPT D ATASET ( BCD ) 7. BCD D EVELOPMENT
Revised: and [Table] two pre-deﬁned concepts, ii) concepts that cannot be explained via auto-labels but are meaningful and interpretable, iii) concepts that are uninterpretable. Figure 6a and 6b are examples of the ﬁrst category mentioned above.

Revised: The former describes a concept composed of different verb forms.

Revised: It does not fully align with any of the ﬁne-grained verb categories but can be deemed as a verb cluster at coarser level.

Revised: Similarly the latter shows a cluster containing singular and plural nouns (described by POS:NN and POS:NNS) and WordNet concepts (WN:noun:food and WN:noun:plants) and can be aligned with a coarser noun category.

Revised: Figure 6c shows a cluster that does not match with any human-deﬁned concept but are interpretable in that the cluster is composed of compound words.

Revised: Lastly in Figure 6d, we show a cluster that we found to be unaligned and uninterpretable.

Revised:  BERT C ONCEPT D ATASET ( BCD )

Original: In addition to the manually labeled dataset, we also developed a large dataset, BCD, using a weakly supervised approach.
Revised: In addition to the manually labeled dataset, we also developed a large dataset, BCD, using a weakly supervised approach.

Original: The motivation of creating such a dataset is to provide a reasonable size dataset that can be effectively use for interpretation studies and other tasks.
Revised: The motivation of creating such a dataset is to provide a reasonable size dataset that can be effectively use for interpretation studies and other tasks.

Original: The idea is to expand the number of tokens for every concept cluster by ﬁnding new token occurrences that belong to the same cluster.
Revised: The idea is to expand the number of tokens for every concept cluster by ﬁnding new token occurrences that belong to the same cluster.

Original: There are several possible ways to develop such a dataset.
Revised: There are several possible ways to develop such a dataset.

Original: Here, we explored two different approaches.
Revised: Here, we explored two different approaches.

Original: First: we compute the centroid for each cluster and ﬁnd the cluster that is closest to the representation of any given new token.
Revised: First: we compute the centroid for each cluster and ﬁnd the cluster that is closest to the representation of any given new token.

Original: However, this did not result in accurate cluster assignment because of the non-spherical and irregularly shaped clusters.
Revised: However, this did not result in accurate cluster assignment because of the non-spherical and irregularly shaped clusters.

Original: Second: we developed a logistic regression classiﬁer (as shown in  Algorithm 2) using our manually annotated dataset to assign new tokens to given concept clusters.
Revised: Second: we developed a logistic regression classiﬁer (as shown in Appendix A, Algorithm 2) using our manually annotated dataset to assign new tokens to given concept clusters.

Original: We found the second approach better than the former in terms of token to cluster assignment.
Revised: We found the second approach better than the former in terms of token to cluster assignment.

Original: We trained the model using 90% of the concept clusters and evaluate its performance on the remaining 10% concept clusters (held-out set).
Revised: We trained the model using 90% of the concept clusters and evaluate its performance on the remaining 10% concept clusters (held-out set).

Original: The classiﬁer achieved a precision score of 75% on the held-out set.
Revised: The classiﬁer achieved a precision score of 75% on the held-out set.

Original: In order to achieve higher precision, we introduce a threshold t = 0 .
Revised: In order to achieve higher precision, we introduce a threshold t = 0 .

Original: 97 on the conﬁdence of the predicted cluster id, assigning new tokens to particular clusters only when the conﬁdence of the classiﬁer is higher than the threshold.
Revised: 97 on the conﬁdence of the predicted cluster id, assigning new tokens to particular clusters only when the conﬁdence of the classiﬁer is higher than the threshold.

Original: This enables better precision in cluster assignment to the new tokens, at the expense of discarding some potentially good assignments, which can be offset by using more data for prediction.
Revised: This enables better precision in cluster assignment to the new tokens, at the expense of discarding some potentially good assignments, which can be offset by using more data for prediction.

Original: Using the trained model we labeled all the tokens from the 2 million random sentences from the unused News 2018 dataset (see Section 4).
Revised: Using the trained model we labeled all the tokens from the 2 million random sentences from the unused News 2018 dataset (see Section 4).

Original: The resulting dataset: BCD is a unique multi-faceted resource consisting of 198 concept labels with a total of 989,424 annotated instances.
Revised: The resulting dataset: BCD is a unique multi-faceted resource consisting of 198 concept labels with a total of 989,424 annotated instances.

Original: The average number of annotated instances per concept label are 4,997.
Revised: The average number of annotated instances per concept label are 4,997.

Original: The utility of this resource is not limited to interpretation, but can serve as a valuable ﬁne-grained dataset for the NLP community at large.
Revised: The utility of this resource is not limited to interpretation, but can serve as a valuable ﬁne-grained dataset for the NLP community at large.

Original: The hierarchy present in the concept labels provides ﬂexibility to use data with various granularity levels.
Revised: The hierarchy present in the concept labels provides ﬂexibility to use data with various granularity levels.

Original: Appendix C provides the detailed statistics for BCD .
Revised: Appendix F provides the detailed statistics for BCD .

Original: 7. C ASE S TUDY
Revised: [Figure] Case Study

Original: In order to showcase the efﬁcacy of BCD as a useful resource in interpretation, we present a casestudy based on neuron-level interpretation in this section.
Revised: In order to showcase the efﬁcacy of BCD as a useful resource in interpretation, we present a case-study based on neuron-level interpretation in this section.

Original: The goal of Neuron-level interpretation in DNNs is to discover a set of neurons responsible for any given concept (Sajjad et al., 2021).
Revised: The goal of Neuron-level interpretation in DNNs is to discover a set of neurons responsible for any given concept (Sajjad et al., 2021).

Original: Most of the work in this direction has identiﬁed neurons responsible for concepts belonging to the classical NLP tasks such as POS, SEM and Chunking.
Revised: Most of the work in this direction has identiﬁed neurons responsible for concepts belonging to the core NLP tasks such as POS, SEM and Chunking.

Original: For instance, (Durrani et al., 2020) showed thata large number of neurons in BERT learn the proper noun (NNP) concept and a relatively fewer number of neurons learn the POS:interjection concept. This kind of analysis provides useful insights into how and where pre-deﬁned concepts are encoded in a network.
Revised: In this work, we found that BERT learns ﬁner categories compared to the ones deﬁned in the core NLP tasks such as proper nouns (NNP).

Original: In this work, we found that BERT splits proper nouns into a large number of ﬁne-grained concepts, each representing a unique facet.
Revised: BCD splits proper nouns into a large number of ﬁne-grained concepts, each representing a unique facet.

Original: We argue that the large number of neurons found for
Revised: We argue that the large number of neurons found for

Original: NNP are the result of it being a very diverse concept, and that there may be neurons that learn speciﬁc aspects of NNP.
Revised: NNP are the result of it being a very diverse concept, and that there may be neurons that learn speciﬁc aspects of NNP.

Original: Neuron analysis for ﬁne-grained concepts (as present in BCD ) such as proper-noun:person:specific-demography would enable a more precise selection of neurons.
Revised: Neuron analysis for ﬁne-grained concepts (as present in BCD ) such as proper-noun:person:specific-demography would enable a more precise selection of neurons.

Original: Aside from the beneﬁt of achieving more insightful analysis, the discovery of speciﬁc neurons will also facilitate various applications such as model control (Bau et al., 2019; Gu et al., 2021;
Revised: Aside from the beneﬁt of achieving more insightful analysis, the discovery of speciﬁc neurons will also facilitate various applications such as model control (Bau et al., 2019; Gu et al., 2021;

Original: Suau et al., 2020), domain adaptation (Gu et al., 2021) and debiasing.
Revised: Suau et al., 2020), domain adaptation (Gu et al., 2021) and debiasing.

Original: In a preliminary experiment, we performed neuron analysis using the PER (person) concept of SEM and the ﬁne-grained concept of Muslim names (Figure 4e).
Revised: In a preliminary experiment, we performed neuron analysis using the PER (person) concept of SEM and the ﬁne-grained concept of Muslim names (Figure 4e).

Original: We then compared the amount of neurons required for each concept.
Revised: We then compared the amount of neurons required for each concept.

Original: For the SEM dataset, we used the standard splits (Appendix B).
Revised: For the SEM dataset, we used the standard splits (Appendix D).

Original: For the ﬁne-grained concept, the BCD dataset consists of 2748 Muslim name instances.
Revised: For the ﬁne-grained concept, the BCD dataset consists of 2748 Muslim name instances.

Original: We randomly selected an equal amount of negative class instances and split the data into 80% train, 10% development and 10% test sets.
Revised: We randomly selected an equal amount of negative class instances and split the data into 80% train, 10% development and 10% test sets.

Original: We used Linguistic Correlation Analysis (Dalvi et al., 2019) and identiﬁed the minimum number of neurons responsible for each concept.
Revised: We used Linguistic Correlation Analysis (Dalvi et al., 2019) and identiﬁed the minimum number of neurons responsible for each concept.

Original: We found that only 19 neurons are enough to represent the ﬁne-grained concept as opposed to 74 neurons for the PER concept of SEM, showing that the BCD dataset enables selection of specialized neurons responsible for very speciﬁc aspects of the language.
Revised: We found that only 19 neurons are enough to represent the ﬁne-grained concept as opposed to 74 neurons for the PER concept of SEM, showing that the BCD dataset enables selection of specialized neurons responsible for very speciﬁc aspects of the language.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have provided a comprehensive analysis of latent concepts learned in the BERT model.
Revised: In this work, we have provided a comprehensive analysis of latent concepts learned in the BERT model.

Original: Our analysis revealed various notable ﬁndings such as, i) the model learns novel concepts which do not strictly adhere to any pre-deﬁned linguistic groups, ii) various latent concepts are composed of multiple independent properties, such as proper-nouns and ﬁrst word in a sentence, iii) lower layers specialize in learning shallow lexical concepts while the middle and higher layers have a better representation of core linguistic and semantic concepts.
Revised: Our analysis revealed various notable ﬁndings such as, i) the model learns novel concepts which do not strictly adhere to any pre-deﬁned linguistic groups, ii) various latent concepts are composed of multiple independent properties, such as proper-nouns and ﬁrst word in a sentence, iii) lower layers specialize in learning shallow lexical concepts while the middle and higher layers have a better representation of core linguistic and semantic concepts.

Original: We also released a novel BERTcentric concept dataset that not only facilitates future interpretation studies, but will also serve as an annotated dataset similar to POS, WordNet and SEM.
Revised: We also released a novel BERTcentric concept dataset that not only facilitates future interpretation studies, but will also serve as an annotated dataset similar to POS, WordNet and SEM.

Original: Different from the classical NLP datasets, it provides a unique multi-facet set of concepts.
Revised: Different from the classical NLP datasets, it provides a unique multi-facet set of concepts.

Original: While this work has led to plenty of insight into the inner workings of BERT and led to a novel resource, increasing the diversity of the observed concepts is an important direction of future work.
Revised: While this work has led to plenty of insight into the inner workings of BERT and led to a novel resource, increasing the diversity of the observed concepts is an important direction of future work.

Original: Working around the memory and algorithm constraints will allow the creation of a more varied concept set.
Revised: Working around the memory and algorithm constraints will allow the creation of a more varied concept set.

Original: Phrase-level concepts is yet another direction that can be explored further.
Revised: Phrase-level concepts is yet another direction that can be explored further.

Original: Finally, a few works on analyzing word representations (both static and contextual) like Reif et al.
Revised: Finally, a few works on analyzing word representations (both static and contextual) like Reif et al.

Original: (2019) and Ethayarajh (2019)
Revised: (2019) and Ethayarajh (2019)

Original: have alluded that taking the principal components into account may lead to a more controlled analysis.
Revised: have alluded that taking the principal components into account may lead to a more controlled analysis.

Original: Incorporating this line of work may yield useful insights into the type of discovered latent clusters.
Revised: Incorporating this line of work may yield useful insights into the type of discovered latent clusters.

Revised: A PPENDIX A C LUSTERING A LGORITHM AND BCD A LGORITHM

Revised: [Table]

Original: A PPENDIX A A NNOTATION D ETAILS
Revised: [Table] B A NNOTATION D ETAILS

Original: For the annotation, we prepared detailed instructions to guide the annotators, which they followed during the annotation tasks.
Revised: For the annotation, we prepared detailed instructions to guide the annotators, which they followed during the annotation tasks.

Original: Our annotation consists of two questions: (i) Q1:
Revised: Our annotation consists of two questions: (i) Q1:

Original: Is the cluster meaningful?
Revised: Is the cluster meaningful?

Original: , (ii) Q2:
Revised: , (ii) Q2:

Original: Can the two clusters be combined to form a meaningful group?.
Revised: Can the two clusters be combined to form a meaningful group?.

Original: The word cluster is represented in a form of a word cloud, where the frequency of the word in the data represents a relative size of the word in the word cloud.
Revised: The word cluster is represented in a form of a word cloud, where the frequency of the word in the data represents a relative size of the word in the word cloud.

Original: To understand the context of each word in the cluster we also facilitated annotators with associated sentences from the dataset.
Revised: To understand the context of each word in the cluster we also facilitated annotators with associated sentences from the dataset.

Original: For the annotation, we provided speciﬁc instructions with examples for each question.
Revised: For the annotation, we provided speciﬁc instructions with examples for each question.

Original: With the second question, the idea is to understand whether two sibling clusters can be combined to form a meaningful super-cluster.
Revised: With the second question, the idea is to understand whether two sibling clusters can be combined to form a meaningful super-cluster.

Original: For the second question, two clusters are siblings, which we automatically identiﬁed from our hierarchical clustering model.
Revised: For the second question, two clusters are siblings, which we automatically identiﬁed from our hierarchical clustering model.

Original: We showed the second question only if a sibling of the main cluster (cluster presented in the ﬁrst question) was available or identiﬁed by the clustering algorithm.
Revised: We showed the second question only if a sibling of the main cluster (cluster presented in the ﬁrst question) was available or identiﬁed by the clustering algorithm.

Original: Hence, annotators did not see the second question in all annotations.
Revised: Hence, annotators did not see the second question in all annotations.

Original: Below we provided instructions which we accompanied with the annotation platform in a form of a tutorial.
Revised: Below we provided instructions which we accompanied with the annotation platform in a form of a tutorial.

Original: A. A NNOTATION I NSTRUCTIONS
Revised: B. A NNOTATION I NSTRUCTIONS

Original: The annotation task was to look ﬁrst look at a group of words (i.e., representing as a cluster) and answer the ﬁrst question.
Revised: The annotation task was to look ﬁrst look at a group of words (i.e., representing as a cluster) and answer the ﬁrst question.

Original: Then, look at two groups of words to answer the second question.
Revised: Then, look at two groups of words to answer the second question.

Original: Q1: I S THE CLUSTER MEANINGFUL ?
Revised: Q1: I S THE CLUSTER MEANINGFUL ?

Original: A word group is meaningful if it contains semantically , syntactically , or lexically similar words.
Revised: A word group is meaningful if it contains semantically , syntactically , or lexically similar words.

Original: The example in Figure 6 has only numbers with two digits, hence, this is a meaningful group.
Revised: The example in Figure 7 has only numbers with two digits, hence, this is a meaningful group.

Original: The labels for this question include the following:
Revised: The labels for this question include the following:

Original: Yes (if represents a meaningful cluster) 2.
Revised: Yes (if represents a meaningful cluster) 2.

Original: No (if it does not represent any meaningful cluster) 3.
Revised: No (if it does not represent any meaningful cluster) 3.

Original: Don’t know or can’t judge (if it does not have enough information to make a judgment.
Revised: Don’t know or can’t judge (if it does not have enough information to make a judgment.

Original: It is recommended to categorize the word groups using this label when the word group is not understandable at all.)
Revised: It is recommended to categorize the word groups using this label when the word group is not understandable at all.)

Original: If the answer to this question is Yes , then the task would be to assign a name to the word cluster using one or more words.
Revised: If the answer to this question is Yes , then the task would be to assign a name to the word cluster using one or more words.

Original: While assigning the name, it is important to maintain the hierarchy.
Revised: While assigning the name, it is important to maintain the hierarchy.

Original: For example, for the above word group in Figure 6, we can assign a name: semantic:number.
Revised: For example, for the above word group in Figure 7, we can assign a name: semantic:number.

Original: This needs to be written in the text box right below this question.
Revised: This needs to be written in the text box right below this question.

Original: While deciding the name of the cluster, the priority has to be to focus on semantics ﬁrst, syntax second, and followed by any other meaningful aspects.
Revised: While deciding the name of the cluster, the priority has to be to focus on semantics ﬁrst, syntax second, and followed by any other meaningful aspects.

Original: While annotating it is also important to consider (i) their relative frequency of the tokens in the cluster, which is clearly visible in the word cluster, (ii) context in a sentence where the word appears in.
Revised: While annotating it is also important to consider (i) their relative frequency of the tokens in the cluster, which is clearly visible in the word cluster, (ii) context in a sentence where the word appears in.

Original: Q2: C AN THE TWO CLUSTERS BE COMBINED TO FORM A MEANINGFUL GROUP ?
Revised: Q2: C AN THE TWO CLUSTERS BE COMBINED TO FORM A MEANINGFUL GROUP ?

Original: For this question, two clusters are shown and the task is to see if they can form a meaningful supercluster after combining.
Revised: For this question, two clusters are shown and the task is to see if they can form a meaningful supercluster after combining.

Original: In these two clusters, the left one is the same cluster annotated for the ﬁrst question.
Revised: In these two clusters, the left one is the same cluster annotated for the ﬁrst question.

Original: The answer (i.e., labels) of this question is similar to Q1:
Revised: The answer (i.e., labels) of this question is similar to Q1:

Original: Yes , No and Don’t know or can’t judge .
Revised: Yes , No and Don’t know or can’t judge .

Original: Depending on the answer to this question the task is to provide a meaningful name similar to
Revised: Depending on the answer to this question the task is to provide a meaningful name similar to

Original: Q1.
Revised: Q1.

Original: In Figure7 and 8  we provide screenshots for Q1 and Q2, respectively.
Revised: In Figure 8 and 9 we provide screenshots for Q1 and Q2, respectively.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: A. A NNOTATION A GREEMENT
Revised: B. A NNOTATION A GREEMENT

Original: We computed annotation agreement using Fleiss κ Fleiss et al.
Revised: We computed annotation agreement using Fleiss κ Fleiss et al.

Original: , Krippendorff’s α Krippendorff (1970) and average observed agreement Fleiss et al.
Revised: , Krippendorff’s α Krippendorff (1970) and average observed agreement Fleiss et al.

Original: In Table 2, we present the annotation agreement with different approaches.
Revised: In Table [Table] we present the annotation agreement with different approaches.

Original: A. C ONCEPT L ABELS
Revised: B. C ONCEPT L ABELS

Original: Out annotation process resultant in 183 concept labels.
Revised: Out annotation process resultant in 183 concept labels.

Original: In Table 3, 4 and 5 we report LEX, POS and
Revised: In Table 3, 4 and 5 we report LEX, POS and

Original: SEM concepts labels, respectively.
Revised: SEM concepts labels, respectively.

Original: B P RE - DEFINED C ONCEPTS I NFORMATION B. P RE - DEFINED C ONCEPTS
Revised: C A NALYSIS

Original: Table 6 provides a list of all pre-deﬁned concepts used in this work. B. D ATA AND E XPERIMENTAL S ETUP We used standard splits for training, development and test data for the 4 linguistic tasks (POS,

Original: SEM, Chunking and CCG super tagging).

Original: The splits to preprocess the data are available through git repository 4 released with Liu et al.

Original: See Table 7 for statistics. We obtained the understudied pre-trained models from the authors of the paper, through personal communication. [Table] named entity:person SEM:entertainment
Revised: Figure 10 shows example concepts.

Original: SEM:ﬁnancial

Original: SEM:named entity:person:ﬁrst name SEM:entertainment:actors

Original: SEM:ﬁnancial:money ﬁgure

Original: SEM:named entity:person:initial SEM:entertainment:ﬁctional

Original: SEM:ﬁnancial:stock

Original: SEM:named entity:person:last name SEM:entertainment:ﬁctional:games of thrones SEM:food:wine related

Original: SEM:negative SEM:entertainment:ﬁlm:hollywood SEM:food and plant SEM:number SEM:entertainment:ﬁlm and tv SEM:gender:feminine
Revised: Figure 11 shows examples of LIWC and WordNet concepts found in lowe layers.

Original: SEM:number:alpha numeric SEM

Original: [Table]
Revised: [Table]

Original: entertainment:gaming

Original: SEM:geopolitics

Original: SEM:number:cardinal:spelled SEM:entertainment:music

Original: SEM

Original: [Table]
Revised: [Table]

Original: group:student

Original: SEM
Revised: SEM

Original: [Table]

Original: number:ﬂoating point SEM

Original: [Table]

Original: entertainment:music:classical

Original: [Table]

Original: :historic:medieval

Original: [Table]

Original: number

Original: [Table]

Original: than hundred

Original: [Table]

Original: SEM:number:one hundred to ﬁve hundredSEM
Revised: SEM:number:one hundred to ﬁve hundred

Original: [Table]

Original: north america [Table] mexico
Revised: SEM:origin:north america:mexico

Original: SEM:number [Table] ordinal [Table]
Revised: SEM:number:ordinal

Original: america [Table] california SEM
Revised: SEM:origin:north america:usa:california

Original: [Table]

Original: number [Table] percentage [Table]
Revised: SEM:number:percentage

Original: polynesia
Revised: SEM:origin:polynesia

Original: SEM:number:quantity:concrete
Revised: SEM:number:quantity:concrete

Original: [Table]

Original: portuguese SEM
Revised: SEM:origin:portuguese

Original: [Table]

Original: number:quantity:vague
Revised: SEM:number:quantity:vague

Original: [Table] brazil SEM [Table]
Revised: SEM:origin:south america:brazil

Original: number:whole [Table]
Revised: SEM:number:whole

Original:  italianSEM:organization
Revised: SEM:origin:spanish italian

Original: [Table]

Original: government SEM [Table]
Revised: SEM:organization:government

Original: polarity
Revised: SEM:polarity

Original: SEM:organization:government:foreign affairs
Revised: SEM:organization:government:foreign affairs

Original: SEM:renovation-related
Revised: SEM:renovation-related

Original: SEM:organization:social ngoSEM
Revised: SEM:organization:social ngo

Original: [Table] science [Table] biology:micro biology
Revised: SEM:science:biology:micro biology

Original: SEM:origin:africa
Revised: SEM:origin:africa

Original: [Table] chemistry
Revised: SEM:science:chemistry

Original: SEM:origin:asia:arab
Revised: SEM:origin:asia:arab

Original: SEM:science:chemistry:material
Revised: SEM:science:chemistry:material

Original: SEM:origin:asia:east asianSEM
Revised: SEM:origin:asia:east asian

Original: [Table]

Original: science [Table] geology
Revised: SEM:science:geology

Original: SEM:origin:asia:indochina:french colony
Revised: SEM:origin:asia:indochina:french colony

Original: SEM:sea related
Revised: SEM:sea related

Original: SEM:origin:asia:myanmarSEM
Revised: SEM:origin:asia:myanmar

Original: [Table]

Original: service [Table] industry
Revised: SEM:service-industry

Original: SEM:origin:asia:south east asian
Revised: SEM:origin:asia:south east asian

Original: SEM:technology
Revised: SEM:technology

Original: SEM:origin:asia:subcontinent
Revised: SEM:origin:asia:subcontinent

Original: SEM:technology:communication
Revised: SEM:technology:communication

Original: SEM:origin:europe
Revised: SEM:origin:europe

Original: SEM:technology:electronic
Revised: SEM:technology:electronic

Original: SEM:origin:europe:belgium and netherlands
Revised: SEM:origin:europe:belgium and netherlands

Original: SEM:technology:electronic:storage-devices:removable
Revised: SEM:technology:electronic:storage-devices:removable

Original: SEM:origin:europe:dutch
Revised: SEM:origin:europe:dutch

Original: SEM:time
Revised: SEM:time

Original: SEM:origin:europe:
Revised: SEM:origin:europe:

Original: france
Revised: france

Original: SEM:time:anniversary descriptors
Revised: SEM:time:anniversary descriptors

Original: SEM:origin:europe:germany
Revised: SEM:origin:europe:germany

Original: SEM:time:century:18th and 19th
Revised: SEM:time:century:18th and 19th

Original: SEM:origin:europe:
Revised: SEM:origin:europe:

Original: germany related
Revised: germany related

Original: SEM:time:months of year
Revised: SEM:time:months of year

Original: SEM:origin:europe:
Revised: SEM:origin:europe:

Original: greece and italy
Revised: greece and italy

Original: SEM:time:timeframe
Revised: SEM:time:timeframe

Original: SEM:origin:europe:north europe
Revised: SEM:origin:europe:north europe

Original: SEM:transportation:way
Revised: SEM:transportation:way

Original: SEM:origin:europe:portugal
Revised: SEM:origin:europe:portugal

Original: SEM:transportation:way:road
Revised: SEM:transportation:way:road

Original: SEM:origin:europe:sweden
Revised: SEM:origin:europe:sweden

Original: SEM:unnamed-entity:role
Revised: SEM:unnamed-entity:role

Original: SEM:origin:europe:switzerland
Revised: SEM:origin:europe:switzerland

Original: SEM:vehicle
Revised: SEM:vehicle

Original: SEM:origin:europe:
Revised: SEM:origin:europe:

Original: uk
Revised: uk

Original: SEM:vehicle:bike
Revised: SEM:vehicle:bike

Original: SEM:origin:french
Revised: SEM:origin:french

Original: SEM:weather
Revised: SEM:weather

Original: SEM:origin:latin america SEM:weather:disaster SEM:origin:latin america related SYN:position:ﬁrst word
Revised: [Table]

Original: SEM:origin:north america:canada

Original: [Figure]
Revised: [Figure]

Original: C BCD D ETAILS
Revised: D P RE - DEFINED C ONCEPTS I NFORMATION D. P RE - DEFINED C ONCEPTS

Original: In Table 8, 9 and
Revised: Table 6 provides a list of all pre-deﬁned concepts used in this work

Original: [Figure]
Revised: [Figure]

Original: we provide concept label with the number of token and type for LEX, POS and 
Revised: [Figure] [Table] D. D ATA AND E XPERIMENTAL S ETUP We used standard splits for training, development and test data for the 4 linguistic tasks (POS,

Original: SEM
Revised: SEM, Chunking and CCG super tagging). The splits to preprocess the data are available through git repository 6 released with Liu et al. (2019a). See Table 7 for statistics. We obtained the understudied pre-trained models from the authors of the paper, through personal communication.

Original: [Figure]
Revised: [Table]

Original: [Table]
Revised: How do models compare with respect to the pre-deﬁned concepts? Similar to BERT, we created layer-wise latent clusters using RoBERTa (Liu et al., 2019b) and XLNet (Yang et al., 2019).

Original: [Table]
Revised: We aligned the clusters with the pre-deﬁned concepts. Figure 12 and 13 shows the results. The overall trend of the models learning shallow concepts in lower layers and linguistic concepts in the middle-higher layers is consistent across all models. A few exceptions are: sufﬁxes have a consistently decreasing pattern for XLNet and RoBERTa, the FirstWord information improved for every higher layer till the last few layers.

Original: [Table]
Revised: Moreover, XLNet showed a signiﬁcant drop in matches with the linguistic tasks for the last layers. The last layers of the models are optimized for the objective function (Kovaleva et al., 2019) and there the drop in matches for most of the concepts reﬂects the existence of novel task-speciﬁc concepts which may not be well aligned with the human-deﬁned concepts. The number of matches as shown in brackets in Figure 5, 12 and 13 provides a relative comparison across models and pre-deﬁned concepts. The RoBERTa and XLNet found substantially more

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: matches with most of the classical NLP tasks compared to BERT. The difference is much more pronounced in the case of POS with RoBERTa have the highest match. This may reﬂect that the latent concepts of RoBERTa follow a closer hierarchy to POS compared to other models. On contrary, for

Revised: WordNet and LIWC BERT showed substantially better alignment than the other two models

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: ETAILS

Revised: In Table 8, 9 and 10, we provide concept label with the number of token and type for LEX, POS and

Revised: SEM.

Original:  244 SEM:technology 4, 149 SEM:technology:communication 4, 463 SEM:technology:communication, SEM:technology:electronic 5, 886 SEM:technology:electronic:storage-devices:removable 21 SEM:time, 21, 526 SEM:time:anniversary descriptors 1, 25 SEM:time:century:18th and 19th 1, 204 SEM:time:months of year 16, 37 SEM:time:timeframe, LEX:sufﬁx:ties 1, 20 SEM:time:timeframe, POS:adjective, LEX:hyphenated 8, 623 SEM:transportation:way, POS:noun, 2, 211 SEM:transportation:way:road, POS:noun, 1, 72 SEM:vehicle, SEM:named entity, POS:proper-noun, SEM:brand 103 SEM:weather, 20 SEM:weather:disaster, 29, 1,631 SYN:
Revised: [Table] [Table] [Table] 244 SEM:technology 4, 149 SEM:technology:communication 4, 463 SEM:technology:communication, SEM:technology:electronic 5, 886 SEM:technology:electronic:storage-devices:removable 21 SEM:time, 21, 526 SEM:time:anniversary descriptors 1, 25 SEM:time:century:18th and 19th 1, 204 SEM:time:months of year 16, 37 SEM:time:timeframe, LEX:sufﬁx:ties 1, 20 SEM:time:timeframe, POS:adjective, LEX:hyphenated 8, 623 SEM:transportation:way, POS:noun, 2, 211 SEM:transportation:way:road, POS:noun, 1, 72 SEM:vehicle, SEM:named entity, POS:proper-noun, SEM:brand 103 SEM:weather, 20 SEM:weather:disaster, 29, 1,631 SYN:

Original: position:ﬁrst word, 7, 1,315 SYN:position:ﬁrst word, 2, 365 SYN:position:ﬁrst word, LEX:case:title case, POS:proper-noun, SEM:named entity:person 1, 342 SYN:position:ﬁrst word, LEX:case:title case, SEM:activism:demonstration 4, 518 SYN:
Revised: position:ﬁrst word, 7, 1,315 SYN:position:ﬁrst word, 2, 365 SYN:position:ﬁrst word, LEX:case:title case, POS:proper-noun, SEM:named entity:person 1, 342 SYN:position:ﬁrst word, LEX:case:title case, SEM:activism:demonstration 4, 518 SYN:

Original: position:ﬁrst word, LEX:case:title case, SEM:weather:disaster 1, 232 SYN:
Revised: position:ﬁrst word, LEX:case:title case, SEM:weather:disaster 1, 232 SYN:

Original: position:ﬁrst word, POS:proper-noun, LEX:case:title case, SEM:named entity:person 2, 879 SYN:position:ﬁrst word, POS:proper-noun, LEX:case:title case, SEM:named entity:person, SEM:origin:asia:east asian, 134 SYN:position:ﬁrst word, SEM:named entity 3, 1,
Revised: position:ﬁrst word, POS:proper-noun, LEX:case:title case, SEM:named entity:person 2, 879 SYN:position:ﬁrst word, POS:proper-noun, LEX:case:title case, SEM:named entity:person, SEM:origin:asia:east asian, 134 SYN:position:ﬁrst word, SEM:named entity 3, 1,

Revised: [Table]