Original: I SOTROPY IN THE C ONTEXTUAL E MBEDDING S PACE : C LUSTERS AND M ANIFOLDS
Revised: I SOTROPY IN THE C ONTEXTUAL E MBEDDING S PACE : C LUSTERS AND M ANIFOLDS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The geometric properties of contextual embedding spaces for deep language models such as BERT and ERNIE, have attracted considerable attention in recent years.
Revised: The geometric properties of contextual embedding spaces for deep language models such as BERT and ERNIE, have attracted considerable attention in recent years

Original: Investigations on the contextual embeddings demonstrate a strong anisotropic space such that most of the vectors fall within a narrow cone, leading to high cosine similarities.
Revised: Investigations on the contextual embeddings demonstrate a strong anisotropic space such that most of the vectors fall within a narrow cone, leading to high cosine similarities.

Original: It is surprising that these LMs are as successful as they are, given that most of their embedding vectors are as similar to one another as they are.
Revised: It is surprising that these LMs are as successful as they are, given that most of their embedding vectors are as similar to one another as they are.

Original: In this paper, we argue that the isotropy indeed exists in the space, from a different but more constructive perspective.
Revised: In this paper, we argue that the isotropy indeed exists in the space, from a different but more constructive perspective.

Original: We identify isolated clusters and low dimensional manifolds in the contextual embedding space, and introduce tools to both qualitatively and quantitatively analyze them.
Revised: We identify isolated clusters and low dimensional manifolds in the contextual embedding space and introduce tools to both qualitatively and quantitatively analyze them.

Original: We hope the study in this paper could provide insights towards a better understanding of the deep language models.
Revised: We hope the study in this paper could provide insights towards a better understanding of the deep language models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider the polysemous English word “bank”. There are two common senses: 1.
Revised: The polysemous English word “bank” has two common senses: 1.

Original: the money sense, a place that people save or borrow money; 2.
Revised: the money sense, a place that people save or borrow money; 2.

Original: the river sense, a slope of earth that prevents the ﬂooding.
Revised: the river sense, a slope of earth that prevents the ﬂooding.

Original: In modern usage, the two senses are very different from one another, though interestingly, both senses share similar etymologies (and both can be traced back to the same word in ProtoGermanic).
Revised: In modern usage, the two senses are very different from one another, though interestingly, both senses share similar etymologies (and both can be traced back to the same word in Proto-Germanic).

Original: In the static embedding , multiple instances of the same word (“bank” in above example) will be represented using the same vector.
Revised: In the static embedding , multiple instances of the same word (e.g. “bank”) will be represented using the same vector.

Original: On the contrary, the contextual embedding assigns different vectors to different instances of the same word, depending on the context.
Revised: On the contrary, the contextual embedding assigns different vectors to different instances of the same word, depending on the context.

Original: Historically, static embedding models like Word2vec (Mikolov et al., 2013b) and GloVe (Pennington et al., 2014), predated contextual embedding models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2018) and ERNIE (Sun et al., 2019).
Revised: Historically, static embedding models like Word2vec (Mikolov et al., 2013b) and GloVe (Pennington et al., 2014), predated contextual embedding models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2018) and ERNIE (Sun et al., 2019).

Original: Much of the literature on language modeling has moved to contextual embeddings recently, largely because of their superior performance onalmost all the downstreaming tasks.
Revised: Much of the literature on language modeling has moved to contextual embeddings recently, largely because of their superior performance on the downstreaming tasks.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: The static embeddings are often found to be easier to interpret.
Revised: The static embeddings are often found to be easier to interpret.

Original: For example, the Word2Vec and
Revised: For example, the Word2Vec and

Original: GloVe papers discuss adding and subtracting vectors, such as: vec(king) - vec(man) + vec(women)
Revised: GloVe papers discuss adding and subtracting vectors, such as: vec(king) - vec(man) + vec(women)

Original: = vec(queen).
Revised: = vec(queen).

Original: Inspired by this relationship, researchers started to explore geometric properties of static embedding spaces.
Revised: Inspired by this relationship, researchers started to explore geometric properties of static embedding spaces.

Original: For example, Mu & Viswanath (2018) proposed a very counter-intuitive method that removes the top principle components (the dominating directions in the transformed embedding space), which surprisingly, improved the representation, as validated on a variety of tasks.
Revised: For example, Mu & Viswanath (2018) proposed a very counter-intuitive method that removes the top principle components (the dominating directions in the transformed embedding space), which surprisingly improved the word representations.

Original: Rather than completely discarding the principle components, Liu et al.
Revised: Rather than completely discarding the principle components, Liu et al.

Original: (2019) proposed to use a technique called Conceptor Negation, to softly suppress transformed dimensions with larger variances.
Revised: (2019) proposed to use a technique called Conceptor Negation, to softly suppress transformed dimensions with larger variances.

Original: Both approaches, simply removing certain principle components as well as Conceptor Negation, produce signiﬁcant improvements over vanilla embeddings obtained by static language models.
Revised: Both approaches, simply removing certain principle components as well as Conceptor Negation, produce signiﬁcant improvements over vanilla embeddings obtained by static language models.

Revised: In Huang et al.

Revised: , the authors studied how to effectively transform static word embeddings from one language to another.

Original: Unfortunately, the strong illustrative representation like the king-queen example above, is no longer obvious in a general contextual embedding space.
Revised: Unfortunately, the strong illustrative representation like the king-queen example above, is no longer obvious in a general contextual embedding space.

Original: Arguing that syntax structure indeed exists in the contextual embeddings, Hewitt & Manning (2019) proposed a structural probe to identify the syntax trees buried in the space, and found the evidence of implicit syntax tree in BERT and ELMo.
Revised: Arguing that syntax structure indeed exists in the contextual embeddings, Hewitt & Manning (2019) proposed a structural probe to identify the syntax trees buried in the space, and found the evidence of implicit syntax tree in BERT and ELMo.

Original: The advantage of contextual embedding over the static counterpart, mainly come from its capability to assign different vectors to the same word, depending on the word sense in the context.
Revised: The advantage of contextual embedding over the static counterpart, mainly come from its capability to assign different vectors to the same word, depending on the word sense in the context.

Original: Researchers in (Reif et al., 2019) found such a geometric representation of word senses in the BERT model.
Revised: Researchers in (Reif et al., 2019) found such a geometric representation of word senses in the BERT model.

Original: These papers reveal the existence of linguistic features embedded implicitly in the contextual vector spaces.
Revised: These papers reveal the existence of linguistic features embedded implicitly in the contextual vector spaces.

Original: The geometric properties of contextual embedding space are also investigated and compared with the static embedding space.
Revised: The geometric properties of contextual embedding space are also investigated and compared with the static embedding space.

Original: Mimno & Thompson (2017) found anisotropy when negative sampling is used.
Revised: Mimno & Thompson (2017) found anisotropy when negative sampling is used.

Original: In (Ethayarajh, 2019), the authors characterize how vectors are distributed in the contextual space.
Revised: In (Ethayarajh, 2019), the authors characterize how vectors are distributed in the contextual space.

Original: They found that most vectors occupy in a relatively narrow cone in the space.
Revised: They found that most vectors occupy in a relatively narrow cone in the space.

Original: Pairs of vectors within this cone have large cosines.
Revised: Pairs of vectors within this cone have large cosines.

Original: This phenomenon can be found in most state-of-the-art contextual embedding models.
Revised: This phenomenon can be found in most state-of-the-art contextual embedding models.

Original: In (Gao et al., 2019), the authors named this phenomenon ”representation degeneration” problem and attempted to mitigate the problem by introducing a regularization term that minimizes cosine similarities between vectors.
Revised: In (Gao et al., 2019), the authors named this phenomenon ”representation degeneration”, and attempted to mitigate the problem by introducing a regularization term that minimizes cosine similarities between vectors.

Original: In a very recent work, Demeter et al.
Revised: In a very recent work, Demeter et al.

Original: (2020) suggest there is a structure weakness in the space that leads to bias when using soft-max, as is common with deep LMs.
Revised: (2020) suggest there is a structure weakness in the space that leads to bias when using soft-max, as is common with deep language models.

Original: 1. M OTIVATION AND C ONTRIBUTIONS
Revised: 1. M OTIVATION AND C ONTRIBUTIONS

Original: Isotropy often makes the space more effectively utilized(all directions could be assigned), and more robust to perturbations (no extreme directions that lead to high condition number).
Revised: Isotropy often makes the space more effectively utilized and more robust to perturbations (no extreme directions that lead to high condition number).

Original: It isvery counter-intuitive and not clear why those contextual embedding models perform remarkably well on many tasks(state-of-the-art performance) given their anisotropic embeddings bring all the vectors close together, hard to distinguish onevector from another.
Revised: It is counter-intuitive and not clear why those contextual embedding models perform remarkably well on many tasks given their anisotropic embeddings bring all the vectors close together, hard to distinguish one from another.

Original: On one hand, it is widely believed that contextual embeddings encode the relevant linguistic information (e.g. (Reif et al., 2019)), but on the other hand, it is also widely believed that the contextual space is anisotropic that representations become degenerated (e.g. (Mimno & Thompson, 2017), (Gao et al., 2019), (Ethayarajh, 2019)).
Revised: On one hand, it is widely believed that contextual embeddings encode the relevant linguistic information (e.g. (Reif et al., 2019)), but on the other hand, it is also widely believed that the contextual space is anisotropic that representations become degenerated (e.g. (Mimno & Thompson, 2017), (Gao et al., 2019), (Ethayarajh, 2019)).

Original: These motivate us to ﬁnd a reasonable understanding that bridges this gap.
Revised: These motivate us to ﬁnd a reasonable understanding that bridges this gap.

Original: This paper is similar in spirit to (Mu & Viswanath, 2018), but different in three aspects.
Revised: This paper is similar in spirit to (Mu & Viswanath, 2018), but different in three aspects.

Original: First, we generalize their work on older static embeddings to more modern contextual embeddings.
Revised: First, we generalize their work on traditional static embeddings to more modern contextual embeddings.

Original: Second, we introduce clustering methods to isolate the space, whereas they used PCA to remove dominant dimensions (that tend to dominate the variance).
Revised: Second, we introduce clustering methods to isolate the space, whereas they used PCA to remove dominant dimensions (that tend to dominate the variance).

Original: Finally, we identify low dimensional manifolds in the space, and introduce an alternative approach (LID) to characterize local subspaces.
Revised: Finally, we identify low dimensional manifolds in the space, and introduce an alternative approach (LID) to characterize local subspaces.

Original: Key Contributions : This paper takes a deeper look into  contextual embedding spaces of popular pre-trained models.
Revised: Key Contributions : This paper takes a deeper look into the contextual embedding spaces of popular pre-trained models.

Original: It identiﬁes the following facts that were misunderstood or not known before: 1)
Revised: It identiﬁes the following facts that were misunderstood or not known before: 1)

Original: We ﬁnd isotropy  in  contextual embedding space, in contrast to previous reports of anisotropy (caused by misleading isolated clusters).
Revised: We ﬁnd isotropy within clusters in the contextual embedding space, in contrast to previous reports of anisotropy (caused by misleading isolated clusters).

Original: We introduce clustering and center shifting to reveal the isotropy, and show more consistent layer-wise behavior across models.
Revised: We introduce clustering and center shifting to reveal the isotropy, and show more consistent layer-wise behavior across models.

Original: We ﬁnd a Swiss-Roll manifold in GPT/GPT2 embeddings, but not in BERT/DistilBERT embeddings.
Revised: We ﬁnd a Swiss-Roll manifold in GPT/GPT2 embeddings, but not in BERT/DistilBERT embeddings.

Original: The manifold is related to word frequency, suggesting a difference in how models evolve as they see more data.
Revised: The manifold is related to word frequency, suggesting a difference in how models evolve as they see more data.

Original: We use approximate Local Intrinsic Dimension (LID) to characterize the manifold, and ﬁnd contextual embedding models, including all BERT, GPT families and ELMo, often have small LIDs.
Revised: We use approximate Local Intrinsic Dimension (LID) to characterize the manifold, and ﬁnd contextual embedding models, including all BERT, GPT families and ELMo, often have small LIDs.

Original: The small LIDs can be viewed as the local anisotropy of the space.
Revised: The small LIDs can be viewed as the local anisotropy of the space.

Revised: The code for this paper could be found at https://github.com/TideDancer/IsotropyContxt .

Original:  A NALYSIS S ETTINGS 2. M ODELS AND D ATASETS
Revised:  A NALYSIS S ETTINGS 2. M ODELS AND D ATASETS

Original: In this paper, we consider popular pre-trained contextual embedding models, including BERT,
Revised: In this paper, we consider popular pre-trained contextual embedding models, including BERT,

Original: DistilBERT (Sanh et al., 2019) (or denoted as D-BERT in the rest of the paper), GPT, GPT2 (Radford et al., 2019) and ELMo.
Revised: DistilBERT (Sanh et al., 2019) (or denoted as D-BERT in the rest of the paper), GPT, GPT2 (Radford et al., 2019) and ELMo.

Original: For the BERT and GPT families, we perform our evaluations on the pretrained uncased base models from Huggingface (https://huggingface.co/transformers/index.html#).
Revised: For the BERT and GPT families, we perform our evaluations on the pretrained uncased base models from Huggingface (https://huggingface.co/transformers/index.html#).

Original: The pre-trained ELMo model is from AllenNLP (https://docs.allennlp.org/v1.0.0/).
Revised: The pre-trained ELMo model is from AllenNLP (https://docs.allennlp.org/v1.0.0/).

Original: BERT and DBERT are non-causal models because of their attention mechanism, where tokens can attend to any token in the input, regardless of their relative positions.
Revised: BERT and DBERT are non-causal models because of their attention mechanism, where tokens can attend to any token in the input, regardless of their relative positions.

Original: In contrast, GPT and GPT2 are causal models because attention is limited to the tokens previously seen in the input.
Revised: In contrast, GPT and GPT2 are causal models because attention is limited to the tokens previously seen in the input.

Original: Different models achieve contextual embedding in different ways.
Revised: Different models achieve contextual embedding in different ways.

Original: For instance, BERT adds positional embeddings to the token embeddings, while ELMo performs vector concatenation.
Revised: For instance, BERT adds positional embeddings to the token embeddings, while ELMo performs vector concatenation.

Original: Most models start with an initial layer that maps token ids to vectors.
Revised: Most models start with an initial layer that maps token ids to vectors.

Original: This paper is not concerned with that lookup table layer and only focus on the layers after that.
Revised: This paper is not concerned with that lookup table layer, and only focuses on the layers after that.

Original: The base BERT, GPT and GPT2 models have 12 layers of interest, indexed from 0 to 11, while D-BERT has 6 layers and ELMo has two.
Revised: The base BERT, GPT and GPT2 models havelayers of interest, indexed from 0 to 11, while D-BERT has 6 layers and ELMo has two.

Original: We use Penn Tree Bank (PTB) (Marcus et al., 1993) and WikiText-2 (Merity et al., 2016) datasets.
Revised: We use Penn Tree Bank (PTB) (Marcus et al., 1993) and WikiText-2 (Merity et al., 2016) datasets.

Original: The PTB has 0.88 million words and WikiText-2 has 2 million.
Revised: The PTB has 0.88 million words and WikiText-2 has 2 million.

Original: Both of them are the standard datasets for language models.
Revised: Both of them are the standard datasets for language models.

Original: In the rest of the paper, we report on PTB since we see similar results with both datasets.
Revised: In the rest of the paper, we report on PTB since we see similar results with both datasets.

Original: Details on WikiText-2 analysis could be found in Appendix.
Revised: Details on WikiText-2 analysis could be found in Appendix.

Original: 2. N OTATION
Revised: 2. N OTATION

Original: For each position in a corpus, we have a word .
Revised: For each position in a corpus, we have a word .

Original: Words are converted into tokens , using the appropriate tokenizer for the model.
Revised: Words are converted into tokens , using the appropriate tokenizer for the model.

Original: Tokenizers could split some words into subwords, therefore, the number of obtained tokens (denoted as n ) could be more than number of words in the corpus.
Revised: Tokenizers could split some words into subwords, therefore, the number of obtained tokens (denoted as n ) could be more than number of words in the corpus.

Original: PTB, for example, contains 0.88 million words, but has n = 1 . 2 million tokens, when processed by BERT’s tokenizer.
Revised: PTB, for example, contains 0.88 million words, but has n = 1 . 2 million tokens, when processed by BERT’s tokenizer.

Original: Let V be the vocabulary , a set of distinct tokens.
Revised: Let V be the vocabulary , a set of distinct tokens.

Original: For any element in the vocabulary V , we call it a type .
Revised: For any element in the vocabulary V , we call it a type .

Original: For example, BERT has a vocabulary of roughly 30 , 000 types.
Revised: For example, BERT has a vocabulary of roughly 30 , 000 types.

Original: We may mix using “word” and “type” for ease of reading.
Revised: We may mix using “word” and “type” for ease of reading.

Original: We denote the i -th type in V as t i .
Revised: We denote the i -th type in V as t i .

Original: Let Φ( t i ) =
Revised: Let Φ( t i ) =

Original: { φ 1 ( t i ) , φ 2 ( t i ) , . . .
Revised: { φ 1 ( t i ) , φ 2 ( t i ) , . . .

Original: } be the set of all embedding instances of t i (note that different contexts in the corpus yield different embeddings of t i ).
Revised: } be the set of all embedding instances of t i (note that different contexts in the corpus yield different embeddings of t i ).

Original: By construction, (cid:80) t | Φ( t ) | = n .
Revised: By construction, (cid:80) t | Φ( t ) | = n .

Original: We deﬁne the inter-type cosine similarity as
Revised: We deﬁne the inter-type cosine similarity as

Original: [Equation]
Revised: [Equation]

Original: where φ ( t i ) is one random sample from Φ( t i ) , and the same for φ ( t j ) ∈
Revised: where φ ( t i ) is one random sample from Φ( t i ) , and the same for φ ( t j ) ∈

Original: Φ( t j ) .
Revised: Φ( t j ) .

Original: The expectation is taken over all pairs of different types.
Revised: The expectation is taken over all pairs of different types.

Original: Similarly, we deﬁne the intra-type cosine similarity as
Revised: Similarly, we deﬁne the intra-type cosine similarity as

Original: [Equation]
Revised: [Equation]

Original: where the inner expectation is over different embeddings φ ( t i ) for the same type t i , and the outer expectation is over all types.
Revised: where the inner expectation is over different embeddings φ ( t i ) for the same type t i , and the outer expectation is over all types.

Original: Note that S inter and S intra take values between − 1 and 1 .
Revised: Both S inter and S intra take values between − 1 and 1 .

Original: Note that for i.i.d.
Revised: Note that for i.i.d.

Original: Gaussian random samples x, y , the expected cosine similarity E [cos( x, y )]
Revised: Gaussian random samples x, y , the expected cosine similarity E [cos( x, y )]

Original: = 0 .
Revised: = 0 .

Original: A cosine value closer to 0 often indicates strong isotropy.
Revised: A cosine value closer to 0 often indicates strong isotropy.

Original: Clearly, the inter-type metric describes the similarity between different types, where the intra-type one measures similarity between same type’s embedding instances.
Revised: Clearly, the inter-type metric describes the similarity between different types, where the intra-type one measures similarity between same type’s embedding instances.

Original: Our deﬁnitions of S inter and
Revised: Our deﬁnitions of S inter and

Original: S intra are similar to the measures used in Ethayarajh (2019), but at the corpus level.
Revised: S intra are similar to the measures used in Ethayarajh (2019), but at the corpus level.

Original: Note that some types are more frequent than others, especially under a Zipﬁan distribution (Piantadosi, 2014), and therefore, the size of Φ( t ) varies dramatically with the frequency of  t .
Revised: Note that some types are more frequent than others, especially under a Zipﬁan distribution (Piantadosi, 2014), and therefore, the size of Φ( t ) varies dramatically with the frequency of type t .

Original: 2. A N INITIAL LOOK AT ANISOTROPY
Revised: 2. A N INITIAL LOOK AT ANISOTROPY

Original: Inspired by Ethayarajh (2019), we follow their procedure and take a ﬁrst look at the anisotropy identiﬁed by Mimno & Thompson (2017) and Ethayarajh (2019), in the contextual embedding space.
Revised: Inspired by Ethayarajh (2019), we follow their procedure and take a ﬁrst look at the anisotropy identiﬁed by Mimno & Thompson (2017) and Ethayarajh (2019), in the contextual embedding space.

Original: [Figure]
Revised: [Figure]

Original: Figure 1 shows strong anisotropy effects in a number of models.
Revised: Figure 1 shows strong anisotropy effects in a number of models.

Original: These ﬁndings are consistent with
Revised: These ﬁndings are consistent with

Original: Ethayarajh (2019), though we use slightly different metrics.
Revised: Ethayarajh (2019), though we use slightly different metrics.

Original: The plots show expected cosine ( S inter and S intra ) as a function of layer.
Revised: The plots show expected cosine ( S inter and S intra ) as a function of layer.

Original: For efﬁciency, we approximate S intra by imposing a limit of 1, samples for frequent types, t , if | Φ( t ) | > 1000 .
Revised: For efﬁciency, we approximate S intra by imposing a limit of 1, samples for frequent types, t , if | Φ( t ) | > 1000 .

Original: From the ﬁgure we can see the following:
Revised: From the ﬁgure we can see the following:

Original: • Both S inter and S intra are high ( (cid:29) 0 ) across almost all the layers and all the models.
Revised: • Both S inter and S intra are high ( (cid:29) 0 ) across almost all the layers and all the models.

Original: In particular, the same as reported in Ethayarajh (2019), GPT2 is relatively more anisotropic.
Revised: In particular, the same as reported in Ethayarajh (2019), GPT2 is relatively more anisotropic.

Original: • S inter tends to increase with layer, in contrast with S intra which in general decreases but with ﬂuctuations.
Revised: • S inter tends to increase with layer, in contrast with S intra which in general decreases but with ﬂuctuations.

Original: This means that embeddings for different types are moving closer to one another at deeper layers, while embeddings for the same type’s instances are spreading away.
Revised: This means that embeddings for different types are moving closer to one another at deeper layers, while embeddings for the same type’s instances are spreading away.

Original: • The last layer is often special.
Revised: • The last layer is often special.

Original: Note that the last layer has smaller cosines than the second last in most cases, with the notable exception of GPT2.
Revised: Note that the last layer has smaller cosines than the second last in most cases, with the notable exception of GPT2.

Original: In summary, we observe large cosines (across layers/models), especially for the GPT2 model.
Revised: In summary, we observe large cosines (across layers/models), especially for the GPT2 model.

Original: When cosines are close to 1 , embeddings lie in a subspace deﬁned by a very narrow cone (Ethayarajh, 2019).
Revised: When cosines are close to 1 , embeddings lie in a subspace deﬁned by a very narrow cone (Ethayarajh, 2019).

Original: One might expect embeddings to be more effective if they took advantage of a larger subspace.
Revised: One might expect embeddings to be more effective if they took advantage of a larger subspace.

Original: Are these models missing an opportunity to have the beneﬁts from isotropy (Mu & Viswanath, 2018)?
Revised: Are these models missing an opportunity to have the beneﬁts from isotropy (Mu & Viswanath, 2018)?

Original: We answer this question in the following sections.
Revised: We answer this question in the following sections.

Original:  C LUSTERS IN THE E MBEDDING S PACE 3. E FFECTIVE D IMENSIONS
Revised:  C LUSTERS IN THE E MBEDDING S PACE 3. E FFECTIVE D IMENSIONS

Original: There are m = 768 embedding dimensions for BERT, D-BERT, GPT and GPT2, and m =dimensions for ELMo.
Revised: There are m = 768 embedding dimensions for BERT, D-BERT, GPT and GPT2, and m =dimensions for ELMo.

Original: We perform PCA to reduce the number of dimensions from m down to k .
Revised: We perform PCA to reduce the number of dimensions from m down to k .

Original: For each layer of each model, we start with the data matrix, M ∈ R n × m , where n is the number of input tokens ( n = 1 .
Revised: For each layer of each model, we start with the data matrix, M ∈ R n × m , where n is the number of input tokens ( n = 1 .

Original: 2 M for PTB dataset), and m is the original number of dimensions.
Revised: 2 M for PTB dataset), and m is the original number of dimensions.

Original: After PCA, we end up with a smaller matrix, ˆ M ∈ R n × k .
Revised: After PCA, we end up with a smaller matrix, ˆ M ∈ R n × k .

Original: Let the explained variance ratio be: r k = (cid:80) k − 1 i =

Revised: Let the explained variance ratio be: r k = (cid:80) k − 1 i =

Original: σ i / (cid:80) m − 1 i =

Revised: σ i / (cid:80) m − 1 i =

Original: σ i , where σ i is the i -th largest eigen value of M ’s covariance matrix.
Revised: σ i , where σ i is the i -th largest eigen value of M ’s covariance matrix.

Original: In this way, we deﬁne the (cid:15) -effective-dimension to be: d ( (cid:15) )
Revised: In this way, we deﬁne the (cid:15) -effective-dimension to be: d ( (cid:15) )

Original: (cid:44) arg min k r k ≥ (cid:15) .
Revised: (cid:44) arg min k r k ≥ (cid:15) .

Original: For example, d (0 .
Revised: For example, d (0 .

Original: 8) =means that 2 dimensions capture 80% of the variance.
Revised: 8) =means that 2 dimensions capture 80% of the variance.

Original: There is a direct connection between d and isotropy: the larger d often implies more isotropy, as data spreads in multiple dimensions.
Revised: There is a direct connection between d and isotropy: the larger d often implies more isotropy, as data spreads in multiple dimensions.

Original: [Table]
Revised: [Table]

Original: BERT           270 D-BERT     175 GPT           307 GPT            1 ELMo
Revised: BERT           270 D-BERT     175 GPT           307 GPT            1 ELMo

Original: Table 1 reports d (0 .
Revised: Table 1 reports d (0 .

Original: 8) for different layers and models.
Revised: 8) for different layers and models.

Original: It is surprising that GPT2 has so few effective dimensions, especially, d (0 .
Revised: It is surprising that GPT2 has so few effective dimensions, especially, d (0 .

Original: 8) = 1 for layer 2 to 6.
Revised: 8) = 1 for layer 2 to 6.

Original: The surprisingly small effective dimensionality is another way of saying that GPT2 vectors fall in a narrow cone, and consequently, their pairwise cosines are large.
Revised: The surprisingly small effective dimensionality is another way of saying that GPT2 vectors fall in a narrow cone, and consequently, their pairwise cosines are large.

Original: If all the vectors lie on a 1-D line, all the cosines would be 1, and there would be hardly any model capacity.
Revised: If all the vectors lie on a 1-D line, all the cosines would be 1, and there would be hardly any model capacity.

Original: These observations motivate us to look deeper into the embedding space.
Revised: These observations motivate us to look deeper into the embedding space.

Original: 3. I SOLATED CLUSTERS
Revised: 3. I SOLATED CLUSTERS

Original: [Figure]
Revised: [Figure]

Original: By performing PCA to project the original data into a 3-D view, we can visualize GPT2’s layer 6’s embedding space in Figure 2a.
Revised: By performing PCA to project the original data into a 3-D view, we can visualize GPT2’s layer 6’s embedding space in Figure 2a.

Original: The three axes refer to the ﬁrst three principle components, which account for 82 .
Revised: The three axes refer to the ﬁrst three principle components, which account for 82 .

Original: 8% of the total variance.
Revised: 8% of the total variance.

Original: All the explained variance ratio will be reported throughout the rest of the paper.
Revised: All the explained variance ratio will be reported throughout the rest of the paper.

Original: The axes values are raw coordinates after PCA.
Revised: The axes values are raw coordinates after PCA.

Original: In Figure 2a, there are two disconnected islands that are far away from each other.
Revised: In Figure 2a, there are two disconnected islands that are far away from each other.

Original: Note that the ﬁrst dimension coordinate values spans from 0 to 3000 , signiﬁcantly wider than the other 2 dimensions.
Revised: Note that the ﬁrst dimension coordinate values spans from 0 to 3000 , signiﬁcantly wider than the other 2 dimensions.

Original: In fact this ﬁrst principle dimension dominates the total variance.
Revised: In fact this ﬁrst principle dimension dominates the total variance.

Original: The left island is bigger than the one on the right.
Revised: The left island is bigger than the one on the right.

Original: The fact that the two islands are so well separated by the ﬁrst principle component suggests that classifying points by island membership accounts for much of the variance.
Revised: The fact that the two islands are so well separated by the ﬁrst principle component suggests that classifying points by island membership accounts for much of the variance.

Original: This two-island property is exhibited in layers 2 through 10 for GPT2.
Revised: This two-island property is exhibited in layers 2 through 10 for GPT2.

Original: The two islands merge into a single large cluster in the last layer.
Revised: The two islands merge into a single large cluster in the last layer.

Original: We observe similar clustering behavior for all the models across all the layers, though the separations are less distinct, as illustrated in other panels of Figure 2.
Revised: We observe similar clustering behavior for all the models across all the layers, though the separations are less distinct, as illustrated in other panels of Figure 2.

Original: This is also consistent with Table 1, the less separation, the higher d ( (cid:15) ) values.
Revised: This is also consistent with Table 1, the less separation, the higher d ( (cid:15) ) values.

Original: For GPT2, we had hoped to ﬁnd that some types are associated with one cluster and other types are associated with the other cluster, but that is not veriﬁed in our experiments.
Revised: For GPT2, we had hoped to ﬁnd that some types are associated with one cluster and other types are associated with the other cluster, but that is not veriﬁed in our experiments.

Original: Please refer to the supplementary for visualizations of all layers in all the models.
Revised: Please refer to the supplementary for visualizations of all layers in all the models.

Original: 3. C LUSTERING
Revised: 3. C LUSTERING

Original: Previous literature estimated the space isotropy on pairs of arbitrary tokens, which could reside in two disconnected clusters.
Revised: Previous literature estimated the space isotropy on pairs of arbitrary tokens, which could reside in two disconnected clusters.

Original: But given that the variance is dominated by distances between clusters, such estimation would be biased by the inter-cluster distances.
Revised: But given that the variance is dominated by distances between clusters, such estimation would be biased by the inter-cluster distances.

Original: It is more meaningful to consider a per-cluster investigation rather than a global estimate.
Revised: It is more meaningful to consider a per-cluster investigation rather than a global estimate.

Original: We start by performing clustering on the embedding space.
Revised: We start by performing clustering on the embedding space.

Original: There are many methods for clustering.
Revised: There are many methods for clustering.

Original: We chose K-Means (https://scikit-learn.org/stable/modules/classes.html#), because it is reasonably fast for large inputs ( n = 1 . 2 million vectors) in high ( m ≥ 768 ) dimensions.
Revised: We chose K-Means (https://scikit-learn.org/stable/modules/classes.html#), because it is reasonably fast for large inputs ( n = 1 . 2 million vectors) in high ( m ≥ 768 ) dimensions.

Original: DBSCAN algorithm (Ester et al., 1996) could be an alternative as it is density based, but only works on small dataset.
Revised: DBSCAN algorithm (Ester et al., 1996) could be an alternative as it is density based, but only works on small dataset.

Original: We use the Silhouette method (Rousseeuw, 1987) to determine the number of clusters, | C | .
Revised: We use the Silhouette method (Rousseeuw, 1987) to determine the number of clusters, | C | .

Original: After running
Revised: After running

Original: K-means, each point p (one of the n vectors in M ) is assigned to one of C clusters.
Revised: K-means, each point p (one of the n vectors in M ) is assigned to one of C clusters.

Original: For a data point p assigned to the cluster c ∈ C , calculate the following:
Revised: For a data point p assigned to the cluster c ∈ C , calculate the following:

Original: [Equation]
Revised: [Equation]

Original: q ∈ c, p (cid:54) =
Revised: q ∈ c, p (cid:54) =

Original: q dist ( p, q ) ; b p = min ˜ c (cid:54) =
Revised: q dist ( p, q ) ; b p = min ˜ c (cid:54) =

Original: c (cid:88) q ∈ ˜ c dist ( p, q ) ; s p = (cid:40)
Revised: c (cid:88) q ∈ ˜ c dist ( p, q ) ; s p = (cid:40)

Original: b p − a p max( a p ,b p ) , if | c | >0 , otherwise where a p is the mean distance between p and other points in the same cluster; b p is the minimum (min over ˜ c ) mean distance between p to points of another cluster ˜ c ; and s p is the Silhouette score for point p ∈ c .
Revised: b p − a p max( a p ,b p ) , if | c | >0 , otherwise where a p is the mean distance between p and other points in the same cluster; b p is the minimum (min over ˜ c ) mean distance between p to points of another cluster ˜ c ; and s p is the Silhouette score for point p ∈ c .

Original: The s p takes value ∈ [ − 1 , 1] .
Revised: The s p takes value ∈ [ − 1 , 1] .

Original: The higher s p , the better assignment of p to its cluster.
Revised: The higher s p , the better assignment of p to its cluster.

Original: Better choices of | C | would lead to better values of s p (and better clustering).
Revised: Better choices of | C | would lead to better values of s p (and better clustering).

Original: We deﬁne the MaximumMean-Silhouette (MMS) score for the embedding space as: MMS (cid:44) max different | C | E p [ s p ] , where the maximum is over different | C | values for K-Means.
Revised: We deﬁne the MaximumMean-Silhouette (MMS) score for the embedding space as: MMS (cid:44) max different | C | E p [ s p ] , where the maximum is over different | C | values for K-Means.

Original: Since it is not feasible to evaluating all choices of | C | ∈ [1 , n ] , we consider | C | ∈ [1 , 15] .
Revised: Since it is not feasible to evaluate all choices of | C | ∈ [1 , n ] , we consider | C | ∈ [1 , 15] .

Original: The expectation E p [ s p ] (the mean Silhouette score), is estimated from 20 , 000 sample vectors in M .
Revised: The expectation E p [ s p ] (the mean Silhouette score), is estimated from 20 , 000 sample vectors in M .

Original: We select the best | C | that yields MMS.
Revised: We select the best | C | that yields MMS.

Original: The MMS values provide a systematic way to describe how the clusters are distributed in the space.
Revised: The MMS values provide a systematic way to describe how the clusters are distributed in the space.

Original: If the clusters are very distinct and splitted, this yields a higher MMS.
Revised: If the clusters are very distinct and splitted, this yields a higher MMS.

Original: On the other hand, if clusters are overlapping, blurring together, the MMS score will be low.
Revised: On the other hand, if clusters are overlapping, blurring together, the MMS score will be low.

Original: Note that if MMS < 0 .
Revised: Note that if MMS < 0 .

Original: 1 , we set | C | to be 1 , as the Silhouette score does not show signiﬁcant evidence of more clusters.
Revised: 1 , we set | C | to be 1 , as the Silhouette score does not show signiﬁcant evidence of more clusters.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Table 2 makes it clear that clustering plays an important role in most layers of most models.
Revised: Table 2 makes it clear that clustering plays an important role in most layers of most models.

Original: Some models (BERT and D-BERT) have more clusters, and some have fewer (GPT, GPT2, ELMo).
Revised: Some models (BERT and D-BERT) have more clusters, and some have fewer (GPT, GPT2, ELMo).

Original: This dichotomy of models is also reﬂected in Figure 2.
Revised: This dichotomy of models is also reﬂected in Figure 2.

Original: Maximum-Mean-Silhouette scores are shown in Figure 3.
Revised: Maximum-Mean-Silhouette scores are shown in Figure 3.

Original: There are signiﬁcantly higher MMS values for GPT2, starting from the 2nd layer.
Revised: There are signiﬁcantly higher MMS values for GPT2, starting from the 2nd layer.

Original: Recall in Figure 2a, we showed that two far-away islands exist in the space and their distance dominates the variance.
Revised: Recall in Figure 2a, we showed that two far-away islands exist in the space and their distance dominates the variance.

Original: In Figure 3, the high MMS scores also veriﬁes that.
Revised: In Figure 3, the high MMS scores also veriﬁes that.

Original: Another interesting observation is, for causal models GPT, GPT2 and ELMo, they all have higher MMS in their middle layers but lower MMS in the end.
Revised: Another interesting observation is, for causal models GPT, GPT2 and ELMo, they all have higher MMS in their middle layers but lower MMS in the end.

Original: This means their initial layer and ﬁnal layers’ embeddings tend to merge.
Revised: This means their initial layer and ﬁnal layers’ embeddings tend to merge.

Original: On the contrary, the BERT and DistilBERT have increasing MMS in deeper layers, meaning that the clusters in their embeddings are becoming clearer in deeper layers.
Revised: On the contrary, the BERT and DistilBERT have increasing MMS in deeper layers, meaning that the clusters in their embeddings are becoming clearer in deeper layers.

Original: 3. I SOTROPY IN CENTERED SPACE WITHIN CLUSTERS
Revised: 3. I SOTROPY IN CENTERED SPACE WITHIN CLUSTERS

Original: As suggest by Mu & Viswanath (2018), the embedding space should be measured after shifting the mean to the origin.
Revised: As suggest by Mu & Viswanath (2018), the embedding space should be measured after shifting the mean to the origin.

Original: We do the mea shifting for each cluster, and calculate the adjusted S inter .
Revised: We subtract the mean for each cluster, and calculate the adjusted S inter .

Original: Assuming we have a total of | C | clusters, let
Revised: Assuming we have a total of | C | clusters, let

Original: Φ c ( t ) =
Revised: Φ c ( t ) =

Original: { φ c 1 ( t ) , φ c 2 ( t ) , . . .
Revised: { φ c 1 ( t ) , φ c 2 ( t ) , . . .

Original: } be the set of type t ’s embeddings in cluster c ∈ C , and φ c ( t ) be one random sample in Φ c ( t ) .
Revised: } be the set of type t ’s embeddings in cluster c ∈ C , and φ c ( t ) be one random sample in Φ c ( t ) .

Original: Deﬁne the adjusted similarity:
Revised: Deﬁne the adjusted similarity:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: , where ¯ 
Revised: where ¯ φ

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here E c is the average over different clusters, and ¯ φ c ( t ) is the original embedding shifted by mean (subtract the mean), where the mean is taken over the samples in cluster c .
Revised: Here E c is the average over different clusters, and ¯ φ c ( t ) is the original embedding shifted by mean (subtract the mean), where the mean is taken over the samples in cluster c .

Original: Similarly we deﬁne
Revised: Similarly we deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The Figure 4 illustrates the adjusted cosine similarities S (cid:48) inter and S (cid:48) intra .
Revised: The Figure 4 illustrates the adjusted cosine similarities S (cid:48) inter and S (cid:48) intra .

Original: It reveals that:
Revised: It reveals that:

Original: • For the adjusted inter-type cosine (the left plot), all models are having consistent near-zero S (cid:48) inter .
Revised: • For the adjusted inter-type cosine (the left plot), all models are having consistent near-zero S (cid:48) inter .

Original: This means nearly perfect isotropy exists within each cluster, in each layer of all the models.
Revised: This means nearly perfect isotropy exists within each cluster, in each layer of all the models.

Original: The last layer of GPT2 and BERT has slightly worse isotropic behavior, nevertheless, general inter-type isotropy stays across all layers.
Revised: The last layer of GPT2 and BERT has slightly worse isotropic behavior, nevertheless, general inter-type isotropy stays across all layers.

Original: This reveals the distinguishable embedding vectors.
Revised: This reveals the distinguishable embedding vectors.

Original: • The general decreasing trend of intra-type cosine (the right plot) shows that the multiple instances for the same type/word, is slowly spreading over the layers.
Revised: • The general decreasing trend of intra-type cosine (the right plot) shows that the multiple instances for the same type/word, is slowly spreading over the layers.

Original: This is consistent with the un-centered intra-type cosine shown in Figure 1.
Revised: This is consistent with the un-centered intra-type cosine shown in Figure 1.

Original:  L OW - DIMENSIONAL M ANIFOLDS 4. 
Revised:  L OW - DIMENSIONAL M ANIFOLDS 4.

Original: S WISS R OLL
Revised: S WISS R OLL

Original: MANIFOLD OF GPT/GPT 
Revised: MANIFOLD OF GPT/GPT

Original: While BERT and D-BERT tend to distribute embeddings along more dimensions, GPT and GPT embed tokens in low-dimensional manifolds in their contextual embedding spaces.
Revised: While BERT and D-BERT tend to distribute embeddings along more dimensions, GPT and GPT embed tokens in low-dimensional manifolds in their contextual embedding spaces.

Original: More speciﬁcally, we discover that most of the tokens are embedded on a spiral band, and that band gets thicker in the later layers thereafter form a Swiss Roll shaped surface.
Revised: More speciﬁcally, we discover that most of the tokens are embedded on a spiral band, and that band gets thicker in the later layers thereafter form a Swiss Roll shaped surface.

Original: Figure 5a and 5b show the 2-D front view of the manifold in GPT and GPT2.
Revised: Figure 5a and 5b show the 2-D front view of the manifold in GPT and GPT2.

Original: Note that Figure 5a zooms into the large cluster illustrated in Figure 2a (the left one), and discards the smaller one (the 
Revised: Figure 5a zooms into the large cluster illustrated in Figure 2a (the left one), and discards the smaller one (the right one).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. T OKENS IN THE SPACE
Revised: 4. T OKENS IN THE SPACE

Original: To verify the manifold structure in GPT family we study the token embeddings in the space.
Revised: To verify the manifold structure in GPT family, we study the token embeddings in the space.

Original: It is believed that similar embeddings (e g. the embeddings for two instances of the same word) tend to stay close together in a Euclidean space as they should have high cosine similarities.
Revised: It is believed that similar embeddings (e g. the embeddings for two instances of the same word) tend to stay close together in a Euclidean space, as they should have high cosine similarities.

Original: Figure 7 drills down into the embeddings for six frequent words: three punctuation symbols (“ \ ”, “&”, “.”) and three common words (“the”, “ﬁrst”, “man”).
Revised: Figure 7 drills down into the embeddings for six frequent words: three punctuation symbols (“ \ ”, “&”, “.”) and three common words (“the”, “ﬁrst”, “man”).

Original: Each panel uses four colors: three colors (black, red, green) for three words of interest, plus gold color for all the other tokens.
Revised: Each panel uses four colors: three colors (black, red, green) for three words of interest, plus gold color for all the other tokens.

Original: [Figure]
Revised: [Figure]

Original: manifold structure
Revised: manifold structure

Original: As shown in Figure 7a 7b the BERT model indeed group similar embeddings into small regions in the space (the red, black and green clusters).
Revised: As shown in Figure 7a 7b, the BERT model indeed group similar embeddings into small regions in the space (the red, black and green clusters).

Original: However, the GPT models are assigning similar embeddings along the manifold we observed before.
Revised: However, the GPT models are assigning similar embeddings along the manifold we observed before.

Original: In Figure 7c 7d, the embeddings for the tokens occupy a spiral band that almost cross the entire space.
Revised: In Figure 7c 7d, the embeddings for the tokens occupy a spiral band that almost cross the entire space.

Original: It does not comply with the Euclidean space geometry as points in such a spiral band would not have high cosine similarity.
Revised: It does not comply with the Euclidean space geometry as points in such a spiral band would not have high cosine similarity.

Original: A Riemannian metric must exist, such that the manifold has larger distance between two spiral bands, but smaller distance on the band.
Revised: A Riemannian metric must exist, such that the manifold has larger distance between two spiral bands, but smaller distance on the band.

Original: Note that the 3-D plots are obtained using PCA, so there is no density-based nor non-linear reduction involved.
Revised: Note that the 3-D plots are obtained using PCA, so there is no density-based nor non-linear reduction involved.

Original: Therefore, the manifold structures in GPT embedding spaces are veriﬁed.
Revised: Therefore, the manifold structures in GPT embedding spaces are veriﬁed.

Original: 4. W ORD FREQUENCY
Revised: 4. W ORD FREQUENCY

Original: Another key ﬁnding is that all the models are trying to map the high frequent words/types to some speciﬁc region in the embedding spaces, rather than spreading them out to the entire space.
Revised: Another key ﬁnding is that all the models are trying to map the high frequent words/types to some speciﬁc region in the embedding spaces, rather than spreading them out to the entire space.

Original: In
Revised: In

Original: Figure 8, embeddings (upper row) and corresponding word frequencies (bottom row) of GPT’s layerand 9 are shown.
Revised: Figure 8, embeddings ( 8a 8c ) and corresponding word frequencies ( 8b 8d ) of GPT’s layer 8 andare shown.

Original: The darker red denoted higher frequency and blue is lower frequency.
Revised: The darker red denoted higher frequency and blue is lower frequency.

Original: The numbers at the colorbar show the number of occurrence (of a particular word / type).
Revised: The numbers at the colorbar show the number of occurrence (of a particular word / type).

Original: [Figure]
Revised: [Figure]

Original: The Figure 8a 8c are after PCA, and selecting the two most signiﬁcant dimensions.
Revised: The Figure 8a 8c are after PCA, and selecting the two most signiﬁcant dimensions.

Original: From GPT layer 8 to layer 9, as the Swiss Roll becomes taller, more variance is accounted for along the height of the
Revised: From GPT layer 8 to layer 9, as the Swiss Roll becomes taller, more variance is accounted for along the height of the

Original: Swiss Roll.
Revised: Swiss Roll.

Original: Thus, the perspective switches from a front view to a side view when moving to layer 9.
Revised: Thus, the perspective switches from a front view to a side view when moving to layer 9.

Original: Figures 8b and 8d show that the most frequent words appear at the head of the Swiss Roll, followed by bands of less and less frequent words.
Revised: Figures 8b and 8d show that the most frequent words appear at the head of the Swiss Roll, followed by bands of less and less frequent words.

Original: The least frequent words appear at the far end of the Swiss
Revised: The least frequent words appear at the far end of the Swiss

Original: Roll.
Revised: Roll.

Original: This pattern suggests the model distinguishes more frequent from less frequent words.
Revised: This pattern suggests the model distinguishes more frequent from less frequent words.

Original: As the model ﬁnds more and more rare words, it appends them at the end of the Swiss Roll.
Revised: As the model ﬁnds more and more rare words, it appends them at the end of the Swiss Roll.

Original: 4. M ANIFOLD LOCAL INTRINSIC DIMENSION
Revised: 4. M ANIFOLD LOCAL INTRINSIC DIMENSION

Original: Although the original space dimension is 768 (1024 for ELMo), the manifold we observed has a lower intrinsic dimension .
Revised: Although the original space dimension is 768 (1024 for ELMo), the manifold we observed has a lower intrinsic dimension .

Original: It means the data point on the manifold has fewer degrees of freedom to move around.
Revised: It means the data point on the manifold has fewer degrees of freedom to move around.

Original: For example, on a Swiss Roll in a 3-D space, any point can only have 2-D freedom thus the intrinsic dimension is only 2.
Revised: For example, on a Swiss Roll in a 3-D space, any point can only have 2-D freedom thus the intrinsic dimension is only 2.

Original: A recent research on the intrinsic dimension for deep networks could be found at (Ansuini et al., 2019).
Revised: A recent research on the intrinsic dimension for deep networks could be found at (Ansuini et al., 2019).

Original: In this section, we adopt the Local Intrinsic Dimension (LID) that estimates dimension locally with respect to a reference point.
Revised: In this section, we adopt the Local Intrinsic Dimension (LID) that estimates dimension locally with respect to a reference point.

Original: LID is introduced by Houle (2013), and being used in deep learning model characterization recently, e.g. (Ma et al., 2018).
Revised: LID is introduced by Houle (2013), and being used in deep learning model characterization recently, e.g. (Ma et al., 2018).

Original: The LID is often derived using expansion models (Houle et al., 2012), which tries to obtain the local dimension in the vicinity of a reference point from the growth (expansion) characteristics.
Revised: The LID is often derived using expansion models (Houle et al., 2012), which tries to obtain the local dimension in the vicinity of a reference point from the growth (expansion) characteristics.

Original: To illustrate this, we borrow an example from Ma et al.
Revised: To illustrate this, we borrow an example from Ma et al.

Original: Let γ be the radius of an m -D ball in the Euclidean space, denote its volume as ν , then the volume’s growth rate is proportional to γ m , i.e. ν 2 /ν 1 = ( γ 2 /γ 1 ) m , from which we can infer the local dimension ˜ m by ˜ m
Revised: Let γ be the radius of an m -D ball in the Euclidean space, denote its volume as ν , then the volume’s growth rate is proportional to γ m , i.e. ν 2 /ν 1 = ( γ 2 /γ 1 ) m , from which we can infer the local dimension ˜ m by ˜ m

Original: = log( ν 2 /ν 1 ) / log( γ 2 /γ 1 ) .
Revised: = log( ν 2 /ν 1 ) / log( γ 2 /γ 1 ) .

Original: Accurately computing LID is a hard problem which requires a tremendous amount of data samples and enough density around the reference point.
Revised: Accurately computing LID is a hard problem which requires a tremendous amount of data samples and enough density around the reference point.

Original: So fewer-sample estimate of LID is being studied in the past decade.
Revised: So fewer-sample estimate of LID is being studied in the past decade.

Original: One of the efﬁcient estimation is proposed by Amsaleg et al. (2015).
Revised: One of the efﬁcient estimation is proposed by Amsaleg et al. (2015).

Original: This technique relies on K nearest neighbor search ( K -NN).
Revised: This technique relies on K nearest neighbor search ( K -NN).

Original: For a reference point p , denote the set of its K nearest neighbor points as Ψ p = { q 1 , . .
Revised: For a reference point p , denote the set of its K nearest neighbor points as Ψ p = { q 1 , . .

Original: , q K } .
Revised: , q K } .

Original: Then the estimate of LID is computed as:
Revised: Then the estimate of LID is computed as:

Original: [Equation]
Revised: [Equation]

Original: 1 , where the term inside log is the ratio of distance between p to its neighbor, over the maximum distance among them.
Revised: 1 , where the term inside log is the ratio of distance between p to its neighbor, over the maximum distance among them.

Original: In our analysis, we use an efﬁcient nearest neighbor computation package FAISS (Johnson et al., 2017) (https://github.com/facebookresearch/faiss) to perform the K -NN.
Revised: In our analysis, we use an efﬁcient nearest neighbor computation package FAISS (Johnson et al., 2017) (https com ) to perform the K -NN.

Original: We set K = 100 , the same as in (Aum¨uller & Ceccarello, 2019).
Revised: We set K = 100 , the same as in (Aum¨uller & Ceccarello, 2019).

Original: (cid:96) 2 distance is used, i.e. dist ( p, q )
Revised: (cid:96) 2 distance is used, i.e. dist ( p, q )

Original: = (cid:107) p − q (cid:107) 2 .
Revised: = (cid:107) p − q (cid:107) 2 .

Original: We report the mean LID over all the samples  as E p [ ˜ LID ( p )] , in Figure 9.
Revised: We report the mean LID over all the samples p , as E p [ ˜ LID ( p )] , in Figure 9.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: As shown in Figure 9, the mean LIDs for all the models in all the layers are below 12.
Revised: As shown in Figure 9, the mean LIDs for all the models in all the layers are below 12.

Original: The small mean
Revised: The small mean

Original: LID values reveals that the manifold’s intrinsic dimension is relatively low, especially considering that this is a 768-D (1024 for ELMo) embedding space.
Revised: LID values reveals that the manifold’s intrinsic dimension is relatively low, especially considering that this is a 768-D (1024 for ELMo) embedding space.

Original: Since ELMo’s 1024-D is larger than other models 768-D dimension, its LID is also slightly higher than other models as shown in the ﬁgure.
Revised: Since ELMo’s 1024-D is larger than other models 768-D dimension, its LID is also slightly higher than other models as shown in the ﬁgure.

Original: The existence of a low-dimensional embedding is also suggested in (Reif et al., 2019) when they study the BERT embedding geometry.
Revised: The existence of a low-dimensional embedding is also suggested in (Reif et al., 2019) when they study the BERT embedding geometry.

Original: In all the contextual embedding layers, there is a clear trend of increasing LID values.
Revised: In all the contextual embedding layers, there is a clear trend of increasing LID values.

Original: In Figure 9, we can also see a nearly-linear relationship between layer id and LID.
Revised: In Figure 9, we can also see a nearly-linear relationship between layer id and LID.

Original: With deeper and deeper layers in the net, the manifold is diffusing and slowly loses concentration.
Revised: With deeper and deeper layers in the net, the manifold is diffusing and slowly loses concentration.

Original: This would lead to data samples spreading, consistent with Figure 4 (recall that intra-type cosines decrease with depth).
Revised: This would lead to data samples spreading, consistent with Figure 4 (recall that intra-type cosines decrease with depth).

Original: Note that as layer goes deeper, each token embedding is collecting information from context by adding their embeddings (and non-linear transforms concatenated).
Revised: Note that as layer goes deeper, each token embedding is collecting information from context by adding their embeddings (and non-linear transforms concatenated).

Original: This could explain the spreading / expanding of the local subspace, and therefore the LID increases in deeper layers.
Revised: This could explain the spreading / expanding of the local subspace, and therefore the LID increases in deeper layers.

Original: Table 3 compares LIDs for static and contextual embeddings.
Revised: Table 3 compares LIDs for static and contextual embeddings.

Original: The table reports results for three static embeddings: GloVe / GloVe-2M (Pennington et al., 2014), and GNEWS (Mikolov et al., 2013a).
Revised: The table reports results for three static embeddings: GloVe / GloVe-2M (Pennington et al., 2014), and GNEWS (Mikolov et al., 2013a).

Original: Results for static embedding LIDs are based on Aum ¨ uller & Ceccarello (2019).
Revised: Results for static embedding LIDs are based on Aum ¨ uller & Ceccarello (2019).

Original: Following Aum ¨ uller & Ceccarello (2019), we use cosine distance here: dist (cid:48)
Revised: Following Aum ¨ uller & Ceccarello (2019), we use cosine distance here: dist (cid:48)

Original: ( p, q ) = 1 − cos( p, q ) = 1 −
Revised: ( p, q ) = 1 − cos( p, q ) = 1 −

Original: [Table]
Revised: [Table]

Revised: p

Original: [Table]
Revised: [Table]

Original: q (cid:107) 2 .
Revised: q [Table]

Original: Note that estimates for LID using cosines are very close to the estimates using (cid:96) 2 distances.
Revised: Note that estimates for LID using cosines are very close to the estimates using (cid:96) 2 distances.

Original: Table 3 reports averages of LIDs over each model’s layers.
Revised: Table 3 reports averages of LIDs over each model’s layers.

Original: Even though GloVe (Pennington et al., 2014) in Tablehas much fewer embedding dimensions (100-D compared with BERT’s 768-D), the LID is still higher than all of the contextual embedding models.
Revised: Even though GloVe (Pennington et al., 2014) in Tablehas much fewer embedding dimensions (100-D compared with BERT’s 768-D), the LID is still higher than all of the contextual embedding models.

Original: From the table we can ﬁnd that static embedding spaces generally have higher LID than the contextual ones.
Revised: From the table we can ﬁnd that static embedding spaces generally have higher LID than the contextual ones.

Original: This means that the data points are more isotropic in the static embeddings, possibly due to their large vocabularies.
Revised: This means that the data points are more isotropic in the static embeddings, possibly due to their large vocabularies.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: Previous works have reported the strong anisotropy in deep LMs, which failed to explain the superior performance achieved by these models.
Revised: Previous works have reported the strong anisotropy in deep LMs, which is hard to explain the superior performance achieved by these models.

Original: We suggest that the anisotropy is a global view being largely misled by distinct clusters resided in the space.
Revised: We suggest that the anisotropy is a global view, being largely misled by distinct clusters resided in the space.

Original: Our analysis show that it is more constructive to isolate and transform the space to measure the isotropy.
Revised: Our analysis show that it is more constructive to isolate and transform the space to measure the isotropy.

Original: From this view, within the clusters the spaces  have almost perfect isotropy that could explain the large model capacity.
Revised: From this view, within the clusters, the spaces of different models all have nearly perfect isotropy that could explain the large model capacity.

Original: In addition, we investigate the space geometry for different models.
Revised: In addition, we investigate the space geometry for different models.

Original: Our visualization demonstrates a low-dimensional Swiss Roll manifold for GPT and GPT2 embeddings, that has not been reported before.
Revised: Our visualization demonstrates a low-dimensional Swiss Roll manifold for GPT and GPT2 embeddings, that has not been reported before.

Original: The tokens and word frequencies are presented to qualitatively show the manifold structure.
Revised: The tokens and word frequencies are presented to qualitatively show the manifold structure.

Original: We propose to use the approximate LID to quantitatively measure the local subspace, and compared with static embedding spaces.
Revised: We propose to use the approximate LID to quantitatively measure the local subspace, and compared with static embedding spaces.

Original: The results show smaller LID values for the contextual embedding models, which can be seen as a local anisotropy in the space.
Revised: The results show smaller LID values for the contextual embedding models, which can be seen as a local anisotropy in the space.

Original: We hope this line of research could bring a comprehensive geometric view of contextual embedding space, and gain insights on how the embeddings are affected by attention, compression, multilingualism, etc.
Revised: We hope this line of research could bring a comprehensive geometric view of contextual embedding space, and gain insights on how the embeddings are affected by attention, compression, multilingualism, etc.

Original: Therefore the model performance could be further improved based on the ﬁndings.
Revised: Therefore the model performance could be further improved based on the ﬁndings.

Original: S UPPLEMENTARY : F ULL R ESULTS ON PTB AND W IKI T EXT -2 D ATASETS A R ESULTS ON W IKI T EXT -2 D ATASET A. T HE UNADJUSTED INTER AND INTRA COSINE SIMILARITY
Revised: S UPPLEMENTARY : F ULL R ESULTS ON PTB AND W IKI T EXT -2 D ATASETS A R ESULTS ON W IKI T EXT -2 D ATASET A. T HE UNADJUSTED INTER AND INTRA COSINE SIMILARITY

Original: Note that ”dist” in the following legends represents DistilBERT model
Revised: Note that ”dist” in the following legends represents DistilBERT model

Original: (a) Inter-type cosine similarity.
Revised: (a) Inter-type cosine similarity.

Original: As layers goes deeper, inter-type cosine goes higher.
Revised: As layers goes deeper, inter-type cosine goes higher.

Original: All models’ last layer behaves slightly differ- ently.
Revised: All models’ last layer behaves slightly differ- ently.

Original: (b) Intra-type cosine similarity.
Revised: (b) Intra-type cosine similarity.

Original: The intra- type cosine decreases showing the same type’s embedding instances are spreading in deeper layers.
Revised: The intra- type cosine decreases showing the same type’s embedding instances are spreading in deeper layers.

Original: A. T HE CENTER - SHIFTED AND CLUSTERED COSINE SIMILARITY
Revised: A. T HE CENTER - SHIFTED AND CLUSTERED COSINE SIMILARITY

Original: The inter-type and intra-type cosines are adjusted using the proposed center-shifting and clustering methods.
Revised: The inter-type and intra-type cosines are adjusted using the proposed center-shifting and clustering methods.

Original: Now it reﬂects the isotropy in almost all layers in all models.
Revised: Now it reﬂects the isotropy in almost all layers in all models.

Original: (a) The clustered inter-type cosine (center shifted).
Revised: (a) The clustered inter-type cosine (center shifted).

Original: It shows strong isotropy as the average cosine between different types is close to 0 across all layers in all models.
Revised: It shows strong isotropy as the average cosine between different types is close to 0 across all layers in all models.

Original: The GPT2’s last layer still has slighly higher cosine com- pared with others.
Revised: The GPT2’s last layer still has slighly higher cosine com- pared with others.

Original: (b) The clustered intra-type cosine (center shifted).
Revised: (b) The clustered intra-type cosine (center shifted).

Original: The intra-type cosines are much more consistent than the unadjusted counter- part (the cosine decreases nearly monotonically as layers goes deeper).
Revised: The intra-type cosines are much more consistent than the unadjusted counter- part (the cosine decreases nearly monotonically as layers goes deeper).

Original: [Figure] T [Figure] APPROXIMATE L OCAL 
Revised: A. T HE APPROXIMATE L OCAL I NTRINSIC D IMENSIONS

Original: [Figure]

Original: B F ULL V ISUALIZATION - PTB D ATASET B. BERT
Revised: B F ULL V ISUALIZATION - PTB D ATASET B. BERT

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: B. D ISTIL BERT AND ELM O
Revised: B. D ISTIL BERT AND ELM O

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: B. GPT
Revised: B. GPT

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: B. GPT 
Revised: B. GPT

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: C

Original: [Table] [Table] [Table] [Table]
Revised: IKI T EXT -2 D ATASET C. BERT

Original: [Table]
Revised: [Table]

Original: C
Revised: C. D ISTIL BERT

Original: D ATASET C. BERT

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] C. GPT

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C. GPT 
Revised: C. GPT

Original: C. GPT
Revised: [Table]

Original: D A DDITIONAL S TUDIES D. K-M EANS C LUSTERING A CCURACY
Revised: D A DDITIONAL S TUDIES D. K-M EANS C LUSTERING A CCURACY

Original: We use K-Means to perform clustering, which raises two issues here.
Revised: We use K-Means to perform clustering, which raises two issues here.

Original: First, K-Means is very sensitive to initialization, different initialization could leads to different clustering results.
Revised: First, K-Means is very sensitive to initialization, different initialization could leads to different clustering results.

Original: However, note that in our task, we are not seeking for optimal clustering.
Revised: However, note that in our task, we are not seeking for optimal clustering.

Original: Sub-optimal, e.g. treating two overlapping clusters as a big one, is totally ﬁne.
Revised: Sub-optimal, e.g. treating two overlapping clusters as a big one, is totally ﬁne.

Original: To illustrate this, we add another metric, Davies-Boulding (DB) index (Davies & Bouldin, 1979), to show that slightly different K is ﬁne.
Revised: To illustrate this, we add another metric, Davies-Boulding (DB) index (Davies & Bouldin, 1979), to show that slightly different K is ﬁne.

Original: This DB index is the average similarity between each cluster and its closest cluster.
Revised: This DB index is the average similarity between each cluster and its closest cluster.

Original: The value closer to 0, the better clustering is done.
Revised: The value closer to 0, the better clustering is done.

Original: We still search in [2 , 15] , and choose K with the minimum DB index (MDB).
Revised: We still search in [2 , 15] , and choose K with the minimum DB index (MDB).

Original: MDB sometimes gives different K than that by
Revised: MDB sometimes gives different K than that by

Original: MMS metric.
Revised: MMS metric.

Original: If MDB is > 4 , we discard MDB and treat all data as one single cluster.
Revised: If MDB is > 4 , we discard MDB and treat all data as one single cluster.

Original: We provide the comparison of selecting K using MMS (left) and MDB (right) here in Table 4.
Revised: We provide the comparison of selecting K using MMS (left) and MDB (right) here in Table 4.

Original: We can see that for less-distinct clusters, e.g. in BERT, two metric could yield different K values, due to merging or splitting.
Revised: We can see that for less-distinct clusters, e.g. in BERT, two metric could yield different K values, due to merging or splitting.

Original: For very separated clusters, e.g. in GPT2, the two metric agrees.
Revised: For very separated clusters, e.g. in GPT2, the two metric agrees.

Original: We plot the cosines using MDB’s K values, in Figure 21.
Revised: We plot the cosines using MDB’s K values, in Figure 21.

Original: It is similar to Figure 4, which uses slightly different K from
Revised: It is similar to Figure 4, which uses slightly different K from

Original: MMS.
Revised: MMS.

Original: The values are close to 0 indicating isotropy in the center-shifted clusters.
Revised: The values are close to 0 indicating isotropy in the center-shifted clusters.

Original: This means that the procedure to reveal isotropy, is not sensitive to K in K-Means.
Revised: This means that the procedure to reveal isotropy, is not sensitive to K in K-Means.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Another issue is that K-Means implicitly assumes convex clusters, which often does not hold.
Revised: Another issue is that K-Means implicitly assumes convex clusters, which often does not hold.

Original: In fact, it assumes isotropic convex clusters because we simply use (cid:96) 2 distance.
Revised: In fact, it assumes isotropic convex clusters because we simply use (cid:96) 2 distance.

Original: However, density-based clustering such as DBSCAN, is too slow thus cannot handle these datasets (million level).
Revised: However, density-based clustering such as DBSCAN, is too slow thus cannot handle these datasets (million level).

Original: This is a trade-off to use K-Means, and empirical results above show that it is efﬁcient and very useful to distinguish separated clusters.
Revised: This is a trade-off to use K-Means, and empirical results above show that it is efﬁcient and very useful to distinguish separated clusters.

Original: D. C LUSTERS AND W ORDS
Revised: D. C LUSTERS AND W ORDS

Original: We study the tokens and their relationship to the clusters existed in the contextual embedding spaces.
Revised: We study the tokens and their relationship to the clusters existed in the contextual embedding spaces.

Original: We picked some representative tokens to see how they are distributed.
Revised: We picked some representative tokens to see how they are distributed.

Original: We also study the very unique small cluster in GPT2, and how it connects to the main cluster that is far away.
Revised: We also study the very unique small cluster in GPT2, and how it connects to the main cluster that is far away.

Original: We obtain the following observations:
Revised: We obtain the following observations:

Original: • For BERT, high frequent words (e
Revised: • For BERT, high frequent words (e

Original: g. ’the’) stays at one side of the main big cluster, while low-frequent words are at the other side.
Revised: g. ’the’) stays at one side of the main big cluster, while low-frequent words are at the other side.

Original: • For BERT, punctuation are random but occupy distinct islands: ’!’
Revised: • For BERT, punctuation are random but occupy distinct islands: ’!’

Original: is a small cluster close to the main island; ’‘’ and ’ ´ are distinct islands far away; ’?’
Revised: is a small cluster close to the main island; ’‘’ and ’ ´ are distinct islands far away; ’?’

Original: and some others are on the main cluster.
Revised: and some others are on the main cluster.

Original: • For GPT2, almost all single letters (a to z) and mid-to-high frequent would occupy both the left (big) and right (small) islands.
Revised: • For GPT2, almost all single letters (a to z) and mid-to-high frequent would occupy both the left (big) and right (small) islands.

Original: • For GPT2, we didn’t ﬁnd any token that only appears in the right small island.
Revised: • For GPT2, we didn’t ﬁnd any token that only appears in the right small island.

Original: It seems the token in the small island always has mirrors in the left big cluster.
Revised: It seems the token in the small island always has mirrors in the left big cluster.

Original: • For word types, e.g. noun, verb, etc, we didn’t ﬁnd a clear pattern.
Revised: • For word types, e.g. noun, verb, etc, we didn’t ﬁnd a clear pattern.

Original: We suspect word frequency affects more than categories.
Revised: We suspect word frequency affects more than categories.

Original: We provide a few examples.
Revised: We provide a few examples.

Original: Figure 22, 23 show BERT layer 3, Figure 24, 25 show GPT2 layer 3.
Revised: Figure 22, 23 show BERT layer 3, Figure 24, 25 show GPT2 layer 3.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Based on these observations, we have concluded that frequency plays an important role in the token distributions.
Revised: Based on these observations, we have concluded that frequency plays an important role in the token distributions.

Original: High frequent words and low frequent words are often taking opposite sides of the space.
Revised: High frequent words and low frequent words are often taking opposite sides of the space.

Original: This is also revealed in Section 4.3.
Revised: This is also revealed in Section 4.3.

Original: We are yet not clear what causes this, but we suspect it is related to the training process.
Revised: We are yet not clear what causes this, but we suspect it is related to the training process.

Original: During training, high frequent words are updated more times.
Revised: During training, high frequent words are updated more times.

Original: Also, since they are used in many many different context, they play a role as some shared embedding across context.
Revised: Also, since they are used in many many different context, they play a role as some shared embedding across context.

Original: Similar to the XLM model, the shared embedding are often more isotropic and more concentrated.
Revised: Similar to the XLM model, the shared embedding are often more isotropic and more concentrated.

Original: However, this is early hypothesis and due to future research.
Revised: However, this is early hypothesis and due to future research.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D. E MBEDDING OF T RANSLATION L ANGUAGE M ODEL XLM
Revised: D. E MBEDDING OF T RANSLATION L ANGUAGE M ODEL XLM

Original: We also perform analysis and visualization on the XLM model (Conneau & Lample, 2019).
Revised: We also perform analysis and visualization on the XLM model (Conneau & Lample, 2019).

Original: BERT is mask language model (MLM), GPT is causal language model (CLM), and XLM is translation language model (TLM).
Revised: BERT is mask language model (MLM), GPT is causal language model (CLM), and XLM is translation language model (TLM).

Original: We provide visualization of XLM’s 6 layers embeddings here.
Revised: We provide visualization of XLM’s 6 layers embeddings here.

Original: This is on
Revised: This is on

Original: WikiText-2 dataset.
Revised: WikiText-2 dataset.

Original: [Figure]
Revised: [Figure]

Original: We try to establish a systematic view of embedding geometry for different types of deep LMs.
Revised: We try to establish a systematic view of embedding geometry for different types of deep LMs.

Original: We have hypothesis and very preliminary results here.
Revised: We have hypothesis and very preliminary results here.

Original: BERT (an MLM) show spreading clusters, but not very distinct.
Revised: BERT (an MLM) show spreading clusters, but not very distinct.

Original: GPT (an CLM) shows highly separated clusters.
Revised: GPT (an CLM) shows highly separated clusters.

Original: XLM (an TLM) does not demonstrate clustering effect, and the embedding are centered.
Revised: XLM (an TLM) does not demonstrate clustering effect, and the embedding are centered.

Original: One possible explanation for XLM’s behavior, is that this is a multi-lingual model, and the embedding space have to be shared between languages.
Revised: One possible explanation for XLM’s behavior, is that this is a multi-lingual model, and the embedding space have to be shared between languages.

Original: This is forced during the training process of this translation language models.
Revised: This is forced during the training process of this translation language models.

Original: In that case, a single cluster residing in the center, would be a good shared embedding across languages.
Revised: In that case, a single cluster residing in the center, would be a good shared embedding across languages.

Original: However, this is just hypothesis and requires further study on more models.
Revised: However, this is just hypothesis and requires further study on more models.

Original: D. LID ESTIMATION ROBUSTNESS
Revised: D. LID ESTIMATION ROBUSTNESS

Original: We follow (Aum¨uller & Ceccarello, 2019) to choose K = 100 for K-Nearest-Neighbor (K-NN) search for LID approximation, and make a direct comparison with them.
Revised: We follow (Aum¨uller & Ceccarello, 2019) to choose K = 100 for K-Nearest-Neighbor (K-NN) search for LID approximation, and make a direct comparison with them.

Original: It raises the concern that 100 samples might not be enough to effectively estimate the local dimension.
Revised: It raises the concern that 100 samples might not be enough to effectively estimate the local dimension.

Original: We conduct additional experiments here to select K = 200 , 500 , 1000 , and demonstrate that the LID estimation is robust.
Revised: We conduct additional experiments here to select K = 200 , 500 , 1000 , and demonstrate that the LID estimation is robust.

Original: They provide similar LID estimates across all layers, in all the models.
Revised: They provide similar LID estimates across all layers, in all the models.

Original: Though using more samples indeed obtain very slightly higher values of LID (in Figure 27, we can see a little bit up-shifting from left-most plot to the right-most plot).
Revised: Though using more samples indeed obtain very slightly higher values of LID (in Figure 27, we can see a little bit up-shifting from left-most plot to the right-most plot).

Original: This is expected, as less number of samples often tends to under-estimate, and over-smoothing of LID.
Revised: This is expected, as less number of samples often tends to under-estimate, and over-smoothing of LID.

Original: Nevertheless, the LID is still much smaller than the original dimension 768, so using 100 samples is a good trade-off to efﬁciently approximate LID.
Revised: Nevertheless, the LID is still much smaller than the original dimension 768, so using 100 samples is a good trade-off to efﬁciently approximate LID.

Original: [Figure]
Revised: [Figure]

Original: As layer goes deeper, the LID increases.
Revised: As layer goes deeper, the LID increases.

Original: In other words, the local space dimension expands, at a cost of losing density.
Revised: In other words, the local space dimension expands, at a cost of losing density.

Original: For example, the spiral band (1-D) in GPT’s front layer, becomes a Swiss
Revised: For example, the spiral band (1-D) in GPT’s front layer, becomes a Swiss

Original: Roll (2-D), and the roll surface get thickness (3-D), as layer increases.
Revised: Roll (2-D), and the roll surface get thickness (3-D), as layer increases.

Original: But we are not clear about the reasons yet, only suspect that data is spreading as more context info is added in later layers (the embedding for a token in deeper layer is based on summation of all embeddings in the context, due to attention).
Revised: But we are not clear about the reasons yet, only suspect that data is spreading as more context info is added in later layers (the embedding for a token in deeper layer is based on summation of all embeddings in the context, due to attention).

Original: This is due to future study.
Revised: This is due to future study.

Original: D. A BLATION ANALYSIS ON CLUSTERING
Revised: D. A BLATION ANALYSIS ON CLUSTERING

Original: To better study the clustering effect, we conduct experiment that computes the inter-type cosines, on clusterd-only embeddings and clustered plus center-shifted embeddings.
Revised: To better study the clustering effect, we conduct experiment that computes the inter-type cosines, on clusterd-only embeddings and clustered plus center-shifted embeddings.

Original: The following ﬁgure shows
Revised: The following ﬁgure shows

Original: GPT2’s cosine on original embeddings without adjustment (blue), the clustering-only embeddings (orange)
Revised: GPT2’s cosine on original embeddings without adjustment (blue), the clustering-only embeddings (orange),

Original: In the original embedding without adjustment, we see inconsistent behavior in the last layers.
Revised: In the original embedding without adjustment, we see inconsistent behavior in the last layers.

Original: However, if we perform clustering and measure the S inter within the clusters (orange), we can see much more consistent behavior across layers (more ﬂat curve).
Revised: However, if we perform clustering and measure the S inter within the clusters (orange), we can see much more consistent behavior across layers (more ﬂat curve).

Original: This indicates that the clustering effect exists in all the layers, which is also veriﬁed by layer-wise visualization in Appendix B.4 and C.4.
Revised: This indicates that the clustering effect exists in all the layers, which is also veriﬁed by layer-wise visualization in Appendix B.4 and C.4.

Original: Meanwhile, the large values of cosines in the orange curve are expected.
Revised: Meanwhile, the large values of cosines in the orange curve are expected.

Original: Now cosines are only computed within clusters, where those clusters are not at the origin.
Revised: Now cosines are only computed within clusters, where those clusters are not at the origin.

Original: The higher values here, the more concentrated clusters are.
Revised: The higher values here, the more concentrated clusters are.

Original: These indicate that after clustering, the subspace within each cluster are now consistent, across all the layers.
Revised: These indicate that after clustering, the subspace within each cluster are now consistent, across all the layers.

Original: Finally, we shift those clusters to the origin, and get the green curve (values near 0), indicating isotropic cluster shapes.
Revised: Finally, we shift those clusters to the origin, and get the green curve (values near 0), indicating isotropic cluster shapes.

Original: D. P OSITIONAL E NCODING IN THE G EOMETRY
Revised: D. P OSITIONAL E NCODING IN THE G EOMETRY

Original: It is very interesting to investigate whether the positional encoding affects the geometry in the contextual embedding spaces.
Revised: It is very interesting to investigate whether the positional encoding affects the geometry in the contextual embedding spaces.

Original: In particular, since GPT/GPT2 have a unique Swiss-Roll shaped manifold, that is not observed in other models, we look at how the manifold is related to the positional encoding in GPT2.
Revised: In particular, since GPT/GPT2 have a unique Swiss-Roll shaped manifold, that is not observed in other models, we look at how the manifold is related to the positional encoding in GPT2.

Original: Note that we truncate the whole PTB text into 512-length segments, and feed those segments into the models.
Revised: Note that we truncate the whole PTB text into 512-length segments, and feed those segments into the models.

Original: The positional encoding is applied to each 512-length segment.
Revised: The positional encoding is applied to each 512-length segment.

Original: We pick a few punctuation and words, and draw them in the space labeled by their relative positions in their corresponding segments.
Revised: We pick a few punctuation and words, and draw them in the space labeled by their relative positions in their corresponding segments.

Original: The position ID ranges from 0 to 511.
Revised: The position ID ranges from 0 to 511.

Original: We select 4 punctuation, “, ’ & $”, and four words “the ﬁrst super man”, draw them in Figure30.
Revised: We select 4 punctuation, “, ’ & $”, and four words “the ﬁrst super man”, draw them in Figure30.

Original: The color bar on the side indicates the relative position ID in their segment.
Revised: The color bar on the side indicates the relative position ID in their segment.

Original: Darker color is smaller IDs and lighter color is bigger IDs.
Revised: Darker color is smaller IDs and lighter color is bigger IDs.

Original: Clearly, for both punctuation and words, the center of the Swiss-Roll corresponds to lower position IDs, where the other end of the manifold are high IDs.
Revised: Clearly, for both punctuation and words, the center of the Swiss-Roll corresponds to lower position IDs, where the other end of the manifold are high IDs.

Original: Also, the distribution is monotonic.
Revised: Also, the distribution is monotonic.

Original: From the center to the far end, the position ID increases.
Revised: From the center to the far end, the position ID increases.

Original: This suggests that the positional encoding is indeed highly correlated with the Swiss-Roll manifold for
Revised: This suggests that the positional encoding is indeed highly correlated with the Swiss-Roll manifold for

Original: GPT models.
Revised: GPT models.

Original: The reason causing this is deferred to future study.
Revised: The reason causing this is deferred to future study.

Original: Note that this ﬁnding is consistent with that reported in (Reif et al., 2019), where they found that positions of the token matters (tokens take all neighbors’ information indiscriminately, rather than only attending up to their semantic boundaries) in the BERT embedding geometry.
Revised: Note that this ﬁnding is consistent with that reported in (Reif et al., 2019), where they found that positions of the token matters (tokens take all neighbors’ information indiscriminately, rather than only attending up to their semantic boundaries) in the BERT embedding geometry.

Original: We also study the context/semantic inﬂuence of the embeddings in the next subsection
Revised: We also study the context/semantic inﬂuence of the embeddings in the next subsection.

Original: D. C ONTEXT IN THE G EOMETRY
Revised: D. C ONTEXT IN THE G EOMETRY

Original: We also look at how the context information inﬂuences the geometry.
Revised: We also look at how the context information inﬂuences the geometry.

Original: It is more sophisticated to analyze the context, so we pick a few examples to look at their context and corresponding positions in the embedding spaces.
Revised: It is more sophisticated to analyze the context, so we pick a few examples to look at their context and corresponding positions in the embedding spaces.

Original: In particular, we choose the common polysemous words “like” and “interest”, as two examples.
Revised: In particular, we choose the common polysemous words “like” and “interest”, as two examples.

Original: The word “like” often has two different use cases: 1. favor; 2. similar to.
Revised: The word “like” often has two different use cases: 1. favor; 2. similar to.

Original: There are also some ﬁxed phrases such as “would like”.
Revised: There are also some ﬁxed phrases such as “would like”.

Original: The word “interest” has two senses as well: 1.
Revised: The word “interest” has two senses as well: 1.

Original: like to do something; 2.
Revised: like to do something; 2.

Original: the money sense.
Revised: the money sense.

Original: We identiﬁed the target word token (“like” or “interest”), and then print out 5 tokens before and after the target, as the context for illustration in Figure 31.
Revised: We identiﬁed the target word token (“like” or “interest”), and then print out 5 tokens before and after the target, as the context for illustration in Figure 31.

Original: From the ﬁgure we are not able to identify a clear pattern that word sense is correlated with the geometric space.
Revised: From the ﬁgure we are not able to identify a clear pattern that word sense is correlated with the geometric space.

Original: However, this is only inspected by manually checking a few samples.
Revised: However, this is only inspected by manually checking a few samples.

Original: A full statistical analysis should be carried out in the future work.
Revised: A full statistical analysis should be carried out in the future work.

Original: [Figure]
Revised: [Figure]