Original: P ROBING BERT IN H YPERBOLIC S PACES
Revised: P ROBING BERT IN H YPERBOLIC S PACES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings.
Revised: Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings.

Original: Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space.
Revised: Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space.

Original: This work posits a family of geometrically special spaces, the hyperbolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations.
Revised: This work considers a family of geometrically special spaces, the hyperbolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations.

Original: We introduce a Poincar´e probe , a structural probe projecting these embeddings into a Poincar´e subspace with explicitly deﬁned hierarchies.
Revised: We introduce a Poincar´e probe , a structural probe projecting these embeddings into a Poincar ´ e subspace with explicitly deﬁned hierarchies.

Original: We focus on two probing objectives: (a) dependency trees where the hierarchy is deﬁned as head-dependent structures; (b) lexical sentiments where the hierarchy is deﬁned as the polarity of words (positivity and negativity).
Revised: We focus on two probing objectives: (a) dependency trees where the hierarchy is deﬁned as headdependent structures; (b) lexical sentiments where the hierarchy is deﬁned as the polarity of words (positivity and negativity).

Original: We apply our probes on BERT, a typical contextualized embedding model.
Revised: We apply our probes on BERT, a typical contextualized embedding model.

Original: In the syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possibility that the geometry of BERT syntax may not necessarily be Euclidean.
Revised: In a syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possibility that the geometry of BERT syntax may not necessarily be Euclidean.

Original: In the sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings.
Revised: In a sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings.

Original: We demonstrate the ﬁndings with our Poincar ´ e probe via extensive experiments and visualization.
Revised: We demonstrate the ﬁndings with our Poincar ´ e probe via extensive experiments and visualization.

Original:  I NTRODUCTION 
Revised:  I NTRODUCTION Contextualized

Original: Contextualized word representations with pretrained language models have signiﬁcantly advanced
Revised: word representations with pretrained language models have signiﬁcantly advanced

Original: NLP progress (Peters et al., 2018a; Devlin et al., 2019).
Revised: NLP progress (Peters et al., 2018a; Devlin et al., 2019).

Original: Previous works point out that abundant linguistic knowledge implicitly exists in these representations (Belinkov et al., 2017; Peters et al., 2018b;a; Tenney et al., 2019).
Revised: Previous works point out that abundant linguistic knowledge implicitly exists in these representations (Belinkov et al., 2017; Peters et al., 2018b;a; Tenney et al., 2019).

Original: This paper is primarily inspired by Hewitt & Manning (2019) who propose a structural probe to recover dependency trees encoded under squared Euclidean distance in a syntactic subspace.
Revised: This paper is primarily inspired by Hewitt & Manning (2019) who propose a structural probe to recover dependency trees encoded under squared Euclidean distance in a syntactic subspace.

Original: Although being an implicit assumption, there is no strict evidence that the geometry of these syntactic subspaces should be Euclidean, especially under the fact that the
Revised: Although being an implicit assumption, there is no strict evidence that the geometry of these syntactic subspaces should be Euclidean, especially under the fact that the

Original: Euclidean space has intrinsic difﬁculties for modeling trees (Linial et al., 1995).
Revised: Euclidean space has intrinsic difﬁculties for modeling trees (Linial et al., 1995).

Original: We propose to impose and explore different inductive biases for modeling syntactic subspaces.
Revised: We propose to impose and explore different inductive biases for modeling syntactic subspaces.

Original: The hyperbolic space, a special Riemannian space with constant negative curvature, is a good candidate because of its tree-likeness (Nickel & Kiela, 2017; Sarkar, 2011).
Revised: The hyperbolic space, a special Riemannian space with constant negative curvature, is a good candidate because of its tree-likeness (Nickel & Kiela, 2017; Sarkar, 2011).

Original: We adopt a generalized Poincar ´ e
Revised: We adopt a generalized Poincar´e

Original: Ball , a special model of hyperbolic spaces, to construct a Poincar ´ e probe for contextualized embeddings.
Revised: Ball , a special model of hyperbolic spaces, to construct a Poincar´e probe for contextualized embeddings.

Original: Figure 1 (A, B) give an example of a tree embedded in the Poincar´e ball and compare the
Revised: Figure 1 (A, B) give an example of a tree embedded in the Poincar´e ball and compare the

Original: Euclidean counterparts.
Revised: Euclidean counterparts.

Original: Intuitively, the volume of a Poincar´e ball grows exponentially with its radius , which is similar to the phenomenon that the number of nodes of a full tree grows exponentially with its depth .
Revised: Intuitively, the volume of a Poincar´e ball grows exponentially with its radius , which is similar to the phenomenon that the number of nodes of a full tree grows exponentially with its depth .

Original: This would give “enough space” to embed the tree.
Revised: This would give “enough space” to embed the tree.

Original: In the meantime, the volume of the Euclidean ball grows polynomially and thus has less capacity to embed tree nodes. 
Revised: In the meantime, the volume of the Euclidean ball grows polynomially and thus has less capacity to embed tree nodes. [Figure]

Revised: Before going any further it is crutial to differentiate a probe and supervised parser (Hall Maudslay et al., 2020) and ask what makes a good probe.

Revised: Ideally, a probe should correctly recover syntactic information intrinsically contained in the embeddings , rather than being a powerful parser by itself.

Revised: So it is important that the probe should have restricted modeling power but still be sensitive enough to the existence of syntax.

Revised: For embeddings without strong syntax information (e g., randomly initialized word embeddings), a probe should not aim to assign high parsing scores (because this would overestimate the existence of syntax), while a parser aims for high scores no matter how bad the input embeddings are.

Original:  Our work of probing BERT in hyperbolic spaces is exploratory.
Revised: The quality of a probe is deﬁned by its sensitivity to syntax. Our work of probing BERT in hyperbolic spaces is exploratory.

Original: As opposed to the Euclidean syntactic subspaces in Hewitt & Manning (2019), we posit a new possible metric space, the Poincar´e syntactic subspace, which is more effective for recovering syntax.
Revised: As opposed to the Euclidean syntactic subspaces in Hewitt & Manning (2019), we consider the Poincar ´ e syntactic subspace, and show its effectiveness for recovering syntax.

Original: Figure 1 (C) gives an example of the reconstructed dependency tree embedded in the Poincar´e ball.
Revised: Figure 1 (C) gives an example of the reconstructed dependency tree embedded in the Poincar ´ e ball.

Original: Ideally, a probe should correctly recover syntactic information intrinsically contained in the embeddings , rather than being a powerful parser by itself.

Original: So it is important that the probe should have restricted modeling power but still be sensitive enough to the existence of syntax.

Original: In our experiments, we highlight two important observations of our Poincar´e probe: (a) it does not give higher parsing scores to baseline embeddings (which have no syntactic information) than Euclidean probes, meaning that it is not a[Figure] better parser ; (b) it reveals higher parsing scores, especially for deeper trees and longer sentences , than the Euclidean probe with strictly restricted capacity.
Revised: In our experiments, we highlight two important observations of our Poincar ´ e probe: (a) it does not give higher parsing scores to baseline embeddings (which have no syntactic information) than Euclidean probes, meaning that it is not a better parser ; (b) it reveals higher parsing scores, especially for deeper trees, longer edges, and longer sentences , than the Euclidean probe with strictly restricted capacity.

Original: Observation (b) can be interpreted from two perspectives: (1) it indicates that the Poincar ´ e probe might be more sensitive to the existence of deeper syntax ; (2) the structure of syntactic subspaces of BERT could be different than Euclidean, especially for deeper trees.
Revised: Observation (b) can be interpreted from two perspectives: (1) it indicates that the Poincar ´ e probe might be more sensitive to the existence of deeper syntax ; (2) the structure of syntactic subspaces of BERT could be different than Euclidean, especially for deeper trees.

Original: Consequently,  BERT may exhibit stronger modeling power for deeper syntax in some special metric space, in our case, a Poincar ´ e ball.
Revised: Consequently, the Euclidean probe may underestimate the syntactic capability of BERT, and BERT may exhibit stronger modeling power for deeper syntax in some special metric space, in our case, a Poincar´e ball.

Original: To best exploit the inductive bias for hierarchical structures of hyperbolic space, we generalize our
Revised: To best exploit the inductive bias for hierarchical structures of hyperbolic space, we generalize our

Original: Poincar ´ e probe to sentiment analysis.
Revised: Poincar´e probe to sentiment analysis.

Original: We construct a Poincar ´ e sentiment subspace by predicting sentiments of individual words using vector geometry (Figure 1 D).
Revised: We construct a Poincar´e sentiment subspace by predicting sentiments of individual words using vector geometry (Figure 1 D).

Original: We assume two meta representations for the positive and negative sentiments as the roots in the sentiment subspace.
Revised: We assume two meta representations for the positive and negative sentiments as the roots in the sentiment subspace.

Original: The stronger a word’s polarity is, the closer it locates to its corresponding meta embedding.
Revised: The stronger a word’s polarity is, the closer it locates to its corresponding meta embedding.

Original: In our experiments, with clearly different geometric properties, the Poincar ´ e probe shows that BERT encodes sentiments for each word in a very ﬁne-grained way.
Revised: In our experiments, with clearly different geometric properties, the Poincar´e probe shows that BERT encodes sentiments for each word in a very ﬁne-grained way.

Original: We further reveal how the localization of word embeddings may change according to lexically-controlled contextualization, i.e., how different contexts would affect the geometric location of the embeddings in the sentiment subspace.
Revised: We further reveal how the localization of word embeddings may change according to lexically-controlled contextualization, i.e., how different contexts would affect the geometric location of the embeddings in the sentiment subspace.

Original: In summary, we present an exploratory Poincar´e probe to reveal hierarchical linguistic structures encoded in BERT.
Revised: In summary, we present an exploratory Poincar´e probe to reveal hierarchical linguistic structures encoded in BERT.

Original: From a hyperbolic deep learning perspective, our results indicate the possibility of using Poincar ´ e models to construct more sensitive probes.
Revised: From a hyperbolic deep learning perspective, our results indicate the possibility of using Poincar´e models for learning better representations of linguistic hierarchies.

Original: From a linguistic perspective, we reveal linguistic hierarchies encoded in BERT and their geometric properties and posit that BERT may encode linguistic information in special metrics spaces that are not necessarily Euclidean.
Revised: From a linguistic perspective, we reveal linguistic hierarchies encoded in BERT and their geometric prop- erties and posit that BERT may encode linguistic information in special metrics spaces that are not necessarily Euclidean.

Original: We demonstrate the effectiveness of our approach with extensive experiments and visualization.
Revised: We demonstrate the effectiveness of our approach with extensive experiments and visualization.

Original: Our results can beeasily reproduced using the Jupyter Notebooks in supplementary materials.
Revised: Our results can be reproduced at https://github.com/FranxYao/ PoincareProbe .

Original:  R ELATED W ORK Probing BERT
Revised:  R ELATED W ORK Probing BERT

Original: Recently, there are increasing interests in ﬁnding linguistic information encoded in BERT.
Revised: Recently, there are increasing interests in ﬁnding linguistic information encoded in BERT (Rogers et al., 2020).

Original: One typical line of work is the structural probes aiming to reveal how syntax trees are encoded geometrically in BERT embeddings (Hewitt & Manning, 2019; Reif et al., 2019).
Revised: One typical line of work is the structural probes aiming to reveal how syntax trees are encoded geometrically in BERT embeddings (Hewitt & Manning, 2019; Reif et al., 2019).

Original: Our Poincar´e probe generally follows this line and exploits the geometric properties of hyperbolic spaces for modeling trees.
Revised: Our Poincar´e probe generally follows this line and exploits the geometric properties of hyperbolic spaces for modeling trees.

Original: Wefurther note that the goal of syntactic probes is to ﬁnd syntax trees with strictly limited capacity, i.e., a probe should not be a parser (Hewitt & Manning, 2019;
Revised: We note that the goal of syntactic probes is to ﬁnd syntax trees with strictly limited capacity, i.e., a probe should not be a parser (Hewitt & Manning, 2019;

Original: Kim et al., 2020), and strictly follow this restriction in our experiments.
Revised: Kim et al., 2020), and strictly follow this restriction in our experiments.

Original: Other probing tasks consider a variety of linguistic properties, including morphology (Belinkov et al., 2017), word sense (Reif et al., 2019), phrases (Kim et al., 2020), semantic fragments (Richardson et al., 2020), and other aspects of syntax and semantics (Tenney et al., 2019).
Revised: Other probing tasks consider a variety of linguistic properties, including morphology (Belinkov et al., 2017), word sense (Reif et al., 2019), phrases (Kim et al., 2020), semantic fragments (Richardson et al., 2020), and other aspects of syntax and semantics (Tenney et al., 2019).

Original: Our extended Poincar´e probe for sentiment can be viewed as one typical semantic probe that reveals how BERT geometrically encodes word sentiments.
Revised: Our extended Poincar ´ e probe for sentiment can be viewed as one typical semantic probe that reveals how BERT geometrically encodes word sentiments.

Original: Hyperbolic Deep Learning
Revised: Hyperbolic Deep Learning

Original: Recently, methods using hyperbolic geometry have been proposed for several NLP tasks due to its better inductive bias for capturing hierarchical information than Euclidean space.
Revised: Recently, methods using hyperbolic geometry have been proposed for several NLP tasks due to its better inductive bias for capturing hierarchical information than Euclidean space.

Original: Poincar´e embeddings (Nickel & Kiela, 2017) and P OINCAR ´
Revised: Poincar ´ e embeddings (Nickel & Kiela, 2017) and P OINCAR ´

Original: E G LO V E (Tifrea et al., 2019) learn embeddings of hierarchies using Poincar´e models and exhibit impressive results, especially in low dimension.
Revised: E G LO V E (Tifrea et al., 2019) learn embeddings of hierarchies using Poincar ´ e models and exhibit impressive results, especially in low dimension.

Original: These works show the advantages of hyperbolic geometry for modeling trees while we focus on using hyperbolic spaces for probing contextualized embeddings.
Revised: These works show the advantages of hyperbolic geometry for modeling trees while we focus on using hyperbolic spaces for probing contextualized embeddings.

Original: To learn models in hyperbolic spaces, previous works combine the formalism of M¨obius gyrovector spaces with the Riemannian geometry, derive hyperbolic versions of important mathematical operations such as M¨obius matrix-vector multiplication, and use them to build hyperbolic neural networks (Ganea et al., 2018).
Revised: To learn models in hyperbolic spaces, previous works combine the formalism of M¨obius gyrovector spaces with the Riemannian geometry, derive hyperbolic versions of important mathematical operations such as M ¨ obius matrix-vector multiplication, and use them to build hyperbolic neural networks (Ganea et al., 2018).

Original: Riemannian adaptive optimization methods (Bonnabel, 2013; B´ecigneul &
Revised: Riemannian adaptive optimization methods (Bonnabel, 2013; B ´ ecigneul &

Original: Ganea, 2019) are proposed for gradient-based optimization.
Revised: Ganea, 2019) are proposed for gradient-based optimization.

Original: Techniques in these works are used as the infrastructure in this work for training Poincar´e probes.
Revised: Techniques in these works are used as the infrastructure in this work for training Poincar ´ e probes.

Original:  P OINCAR ´ E P ROBE
Revised:  P OINCAR ´ E P ROBE

Original: We begin by reviewing the basics of Hyperbolic Geometry.
Revised: We begin by reviewing the basics of Hyperbolic Geometry.

Original: We follow the notations from Ganea et al.
Revised: We follow the notations from Ganea et al.

Original: A generalized Poincar ´ e ball is a typical model of hyperbolic space, denoted as ( D nc , g D x ) for c > 0 , where D nc = { x ∈ R n | c (cid:107) x (cid:107) 2 < 1 } is a Riemannian manifold, g D x
Revised: A generalized Poincar ´ e ball is a typical model of hyperbolic space, denoted as ( D nc , g D x ) for c > 0 , where D nc = { x ∈ R n | c (cid:107) x (cid:107) 2 < 1 } is a Riemannian manifold, g D x

Original: = ( λ c x ) 2 I n is the metric tensor,
Revised: = ( λ c x ) 2 I n is the metric tensor,

Original: λ c x = 2 / (1 − c (cid:107) x (cid:107) 2 ) is the conformal factor and c is the negative curvature of the hyperbolic space.
Revised: λ c x = 2 / (1 − c (cid:107) x (cid:107) 2 ) is the conformal factor and c is the negative curvature of the hyperbolic space.

Original: We will use the term hyperbolic and Poincar ´ e interchangeably according to the context.
Revised: We will use the term hyperbolic and Poincar´e interchangeably according to the context.

Original: Our Poincar´e probe uses the standard Poincar´e ball D nc with c = 1 .
Revised: Our Poincar´e probe uses the standard Poincar´e ball D nc with c = 1 .

Original: The distance function for x ,
Revised: The distance function for x ,

Original: y ∈ D nc is:
Revised: y ∈ D nc is:

Original: [Equation]
Revised: [Equation]

Original: where ⊕ c denotes the M¨obius addition , the hyperbolic version of the addition operator.
Revised: where ⊕ c denotes the M¨obius addition , the hyperbolic version of the addition operator.

Original: Note that we recover the Euclidean space R n when c → 0 .
Revised: Note that we recover the Euclidean space R n when c → 0 .

Original: Additionally, we use M ⊗ c x to denote the M¨obius matrix-vector multiplication for a linear map M : R n → R m , which is the hyperbolic version of linear transforms.
Revised: Additionally, we use M ⊗ c x to denote the M¨obius matrix-vector multiplication for a linear map M : R n → R m , which is the hyperbolic version of linear transforms.

Original: We use exp c x ( · ) to denote the exponential map , which maps vectors in the tangent space (in our case, a space projected from the BERT embedding space) to the hyperbolic space.
Revised: We use exp c x ( · ) to denote the exponential map , which maps vectors in the tangent space (in our case, a space projected from the BERT embedding space) to the hyperbolic space.

Original: Their closed-form formulas are detailed in Appendix C.
Revised: Their closed-form formulas are detailed in Appendix C.

Original: Our probes consist two simple linear maps P and Q that project BERT embeddings into a Poincar ´ e syntactic/sentiment subspace.
Revised: Our probes consist two simple linear maps P and Q that project BERT embeddings into a Poincar´e syntactic/sentiment subspace.

Original: Formally, let M denote a pretrained language model that produces a sequence of distributed representations h 1: t given a sentence of t words w 1: t .
Revised: Formally, let M denote a pretrained language model that produces a sequence of distributed representations h 1: t given a sentence of t words w 1: t .

Original: We train a linear map P : R n → R k , n being the dimension of contextualized embeddings and k being the probe rank, that projects the distributed representations to the tangent space.
Revised: We train a linear map P : R n → R k , n being the dimension of contextualized embeddings and k being the probe rank, that projects the distributed representations to the tangent space.

Original: Then the exponential map projects the tangent space to the hyperbolic space.
Revised: Then the exponential map projects the tangent space to the hyperbolic space.

Original: In the hyperbolic space, we construct the Poincar ´ e syntactic/sentiment subspace via another linear map Q : R k
Revised: In the hyperbolic space, we construct the Poincar´e syntactic/sentiment subspace via another linear map Q : R k

Original: → R k .
Revised: → R k .

Original: The equations are:
Revised: The equations are:

Original: [Equation]
Revised: [Equation]

Original: Here P maps the original BERT embedding space to the tangent space of the origin of the Poincar´e ball.
Revised: Here P maps the original BERT embedding space to the tangent space of the origin of the Poincar´e ball.

Original: Then exp 0 ( · ) maps the tangent space to the Poincar´e ball 1 .
Revised: Then exp 0 ( · ) maps the tangent space inside the Poincar´e ball 1 .

Original: Consequently, in equation 3 we use the M¨obius matrix-vector multiplication as the linear transformation in the hyperbolic space 2 .
Revised: Consequently, in equation 3 we use the M¨obius matrix-vector multiplication as the linear transformation in the hyperbolic space 2 .

Original:  P ROBING S YNTAX
Revised:  P ROBING S YNTAX

Original: Following Hewitt & Manning (2019), we aim to test if there exists a simple function (probe) that maps BERT embeddings into a metric space  where squared distances between embeddings or squared norms of embeddings approximate tree distances or node depths.
Revised: Following Hewitt & Manning (2019), we aim to test if there exists a hyperbolic subspace transformed from the original BERT embedding space with simple parameterization where squared distances between embeddings or squared norms of embeddings approximate tree distances or node depths, respectively.

Original: The goal of the probe is to[Table] recover syntactic information intrinsically contained in the embeddings .
Revised: The goal of the probe is to recover syntactic information intrinsically contained in the embeddings .

Original: To this end, a probe should not assign high parsing scores to baseline non-contextualized embeddings (otherwise it would become a parser , rather than being a probe ).
Revised: To this end, a probe should not assign high parsing scores to baseline non-contextualized embeddings (otherwise it would become a parser , rather than being a probe ).

Original: So it is crucial for the probe to have restricted modeling power (in our case, two linear transforms P and Q ) but still being sensitive enough for syntactic structures.
Revised: So it is crucial for the probe to have restricted modeling power (in our case, two linear transforms P and Q ) but still being sensitive enough for syntactic structures.

Original: We further test if the Poincar ´ e probe is able to discover more syntactic information for deeper trees due to its intrinsic bias for modeling trees.
Revised: We further test if the Poincar ´ e probe is able to discover more syntactic information for deeper trees due to its intrinsic bias for modeling trees.

Original: Similar to Hewitt & Manning (2019), we use the squared Poincar ´ e distance to recreate tree distances between word pairs and the squared Poincar ´ e distance to the origin to recreate the depth of a word:
Revised: Similar to Hewitt & Manning (2019), we use the squared Poincar ´ e distance to recreate tree distances between word pairs and the squared Poincar ´ e distance to the origin to recreate the depth of a word:

Original: [Equation]
Revised: [Equation]

Original: where d T ( w i , w j ) denotes the distance between word i , j on their dependency tree, i.e., number of edges linking word i to j and d D ( w i ) denotes the depth of word i in the dependency tree.
Revised: where d T ( w i , w j ) denotes the distance between word i , j on their dependency tree, i.e., number of edges linking word i to j and d D ( w i ) denotes the depth of word i in the dependency tree.

Original: For optimization, we use the Adam (Kingma & Ba, 2014) initialized at learning rate 0.001 and train up to 40 epochs.
Revised: For optimization, we use the Adam (Kingma & Ba, 2014) initialized at learning rate 0.001 and train up to 40 epochs.

Original: We decay the learning rate and perform model selection based on the dev loss.
Revised: We decay the learning rate and perform model selection based on the dev loss.

Original: 4. E XPERIMENTAL S ETTINGS
Revised: 4. E XPERIMENTAL S ETTINGS

Original: Our experiments aim to demonstrate that the Poincar´e probe better recovers deeper syntax in BERT without becoming a parser.
Revised: Our experiments aim to demonstrate that the Poincar ´ e probe better recovers deeper syntax in BERT without becoming a parser.

Original: We denote the probes in Hewitt & Manning (2019)
Revised: We denote the probes in Hewitt & Manning (2019)

Original: Euclidean probes and follow their datasets and major baseline models.
Revised: Euclidean probes and follow their datasets and major baseline models.

Original: Speciﬁcally, we use the Penn Treebank dataset (Marcus & Marcinkiewicz, 1993) and reuse the data processing code in Hewitt & Manning (2019) to convert the data format to Stanford Dependency (de Marneffe et al., 2006).
Revised: Speciﬁcally, we use the Penn Treebank dataset (Marcus & Marcinkiewicz, 1993) and reuse the data processing code in Hewitt & Manning (2019) to convert the data format to Stanford Dependency (de Marneffe et al., 2006).

Original: For baseline models, we use (a) ELM O 0: strong character-level word embeddings with no contextual information.
Revised: For baseline models, we use (a) ELM O 0: strong character-level word embeddings with no contextual information.

Original: All probes should not ﬁnd syntax trees from these embeddings.
Revised: All probes should not ﬁnd syntax trees from these embeddings.

Original: (b) L INEAR : leftto-right structured trees that only contain positional information.
Revised: (b) L INEAR : leftto-right structured trees that only contain positional information.

Original: (c) ELM O 1 and BERT*: strong contextualized embeddings with rich syntactic information.
Revised: (c) ELM O 1 and BERT*: strong contextualized embeddings with rich syntactic information.

Original: All probes should accurately recover all parse trees encoded in them.
Revised: All probes should accurately recover all parse trees encoded in them.

Original: Since the goal of probing is to recover syntax trees in a strict notion, we restrict our Poincar´e probe to 64 dimension, i.e. k = 64 for P and Q , which is at the same level, or smaller than the effective rank of the Euclidean probes reported in Hewitt & Manning (2019).
Revised: Since the goal of probing is to recover syntax trees in a strict notion, we restrict our Poincar´e probe to 64 dimension, i.e. k = 64 for P and Q , which is at the same level, or smaller than the effective rank of the Euclidean probes reported in Hewitt & Manning (2019).

Original: We also emphasize that the parameters of our Poincar´e probe are simply two matrices, which is again signiﬁcantly less than a typical deep neural network parser (Dozat & Manning, 2016).
Revised: We also emphasize that the parameters of our Poincar´e probe are simply two matrices, which is again signiﬁcantly less than a typical deep neural network parser (Dozat & Manning, 2016).

Revised: This transformation is theoretically redundent, we use it primarily for numerical stability during optimiza- tion.

Revised: We further note that such optimization stability is still an open problem in hyperbolic deep learning (Mathieu et al., 2019).

Revised: We leave a detailed investigation to future work.

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: To evaluate the tree distance probes, we report: (a) undirected unlabeled attachment scores (UUAS), the scores showing if unlabeled edges are correctly recovered, against the gold undirected tree; (b) distance Spearman Correlation (DSpr), the scores showing how the recovered distances (either
Revised: To evaluate the tree distance probes, we report: (a) undirected unlabeled attachment scores (UUAS), the scores showing if unlabeled edges are correctly recovered, against the gold undirected tree; (b) distance Spearman Correlation (DSpr), the scores showing how the recovered distances (either

Original: Euclidean or Hyperbolic) correlate with gold tree distances.
Revised: Euclidean or Hyperbolic) correlate with gold tree distances.

Original: To evaluate the tree depth probes, we report: (a) norm Spearman Correlation (NSpr), the scores showing how the true depth ordering correlates with the predicted depth ordering; (b) the corretly identiﬁed root nodes (root%), the scores showing to what extent the probes can identify sentence roots.
Revised: To evaluate the tree depth probes, we report: (a) norm Spearman Correlation (NSpr), the scores showing how the true depth ordering correlates with the predicted depth ordering; (b) the corretly identiﬁed root nodes (root%), the scores showing to what extent the probes can identify sentence roots.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Table 1 shows the major results for tree distance and depth probing.
Revised: Table 1 shows the major results for tree distance and depth probing.

Original: We see that Poincar ´ e probes do not give higher scores than Euclidean probes for ELM O 0 and L INEAR , yet recovers higher parsing scores for embeddings with deep contextualization.
Revised: We see that Poincar´e probes do not give higher scores than Euclidean probes for ELM O 0 and L INEAR , yet recovers higher parsing scores for embeddings with deep contextualization.

Original: The results indicate that: (a) Poincar ´ e probes  might be more sensitive to the existence of syntactic information; (b) the syntactic information encoded in contextualized embeddings may be underestimated by the Euclidean probes due to their incapacity of modeling trees.
Revised: The results indicate that: (a) Poincar´e probes [Figure] might be more sensitive to the existence of syntactic information; (b) the syntactic information encoded in contextualized embeddings may be underestimated by the Euclidean probes due to their incapacity of modeling trees.

Original: We further give a detailed analysis of the two probes from various perspectives on the distance task, which is harder than the depth task.
Revised: We further give a detailed analysis of the two probes from various perspectives on the distance task, which is harder than the depth task.

Original: Different BERT BASE Layers
Revised: Different BERT BASE Layers

Original: Figure 2A reports the distance scores of both the Poinca´e probe and Euclidean probe trained on each layer of BERT BASE .
Revised: Figure 2A reports the distance scores of both the Poinca ´ e probe and Euclidean probe trained on each layer of BERT BASE .

Original: The two are consistent with each other and show a similar tendency that syntax primarily exists in the middle layers.
Revised: The two are consistent with each other and show a similar tendency that syntax primarily exists in the middle layers.

Original: Probe Rank
Revised: Probe Rank

Original: Figure 2B reports the distance scores of probes with different rank k for BERTBASE 7.
Revised: Figure 2B reports the distance scores of probes with different rank k for BERTBASE 7.

Original: Similar to Hewitt & Manning (2019), we see that the scores do not increase after 64 dimension with the Poincar´e probe.
Revised: Similar to Hewitt & Manning (2019), we see that the scores do not increase after 64 dimension with the Poincar ´ e probe.

Original: We posit there might be some intrinsic syntactic subspace whose dimension is close to 64, but leave further investigation to future work.
Revised: We posit there might be some intrinsic syntactic subspace whose dimension is close to 64, but leave further investigation to future work.

Original: Sentence and Edge Lengths
Revised: Sentence and Edge Lengths

Original: Figure 2C reports the distance scores of sentences with different lengths for BERT BASE 7.
Revised: Figure 2C reports the distance scores of sentences with different lengths for BERT BASE 7.

Original: We highlight the result that Poincar´e probes give higher scores for longer sentences than Euclidean probes, meaning that they better reconstruct deeper syntax.
Revised: We highlight the result that Poincar´e probes give higher scores for longer sentences than Euclidean probes, meaning that they better reconstruct deeper syntax.

Original: Further more, Figure 3  compares edge distance (length) distributionson the dev set between ground truth edges, and edges predicted by Euclidean probes and Poincar´e probes.
Revised: Further more, Figure 3 (left) compares edge distance (length) distributions between ground truth edges, and edges predicted by Euclidean probes and Poincar´e probes.

Original: The distribution from the Poincar´e probes are closer to the ground truth distribution, especially for longer trees.
Revised: The distribution from the Poincar´e probes are closer to the ground truth distribution, especially for longer trees.

Revised: Figure 3 (right) shows that Poincar´e probes consistently achieve better recall for edge types with longer average length.

Revised: The margin is especially large for “advcl”, “csubj”, and “mark”.

Original: Figure 11 in the Appendix further compares minimum spanning tree results from predicted squared distances on BERT BASE 7 (we randomly sample 12 instances from the dev set).
Revised: Figure 11 in the Appendix further compares minimum spanning tree results from predicted squared distances on BERT BASE 7 (we randomly sample 12 instances from the dev set).

Original: These observations would be[Figure] particularly interesting from a linguistic perspective as the syntactic structure for longer sentences are more complicated and challenging than those for shorter sentences.
Revised: These observations would be particularly interesting from a linguistic perspective as the syntactic structure for longer sentences are more complicated and challenging than those for shorter sentences.

Original: A promising future direction would be using hyperbolic spaces for parsing.
Revised: A promising future direction would be using hyperbolic spaces for parsing.

Original: Curvature of the Hyperbolic Space
Revised: Curvature of the Hyperbolic Space

Original: We further characterize the structure of the hyperbolic syntactic subspace with the curvature parameter, which measures how “curved” the space is (Figure 2D).
Revised: We further characterize the structure of the hyperbolic syntactic subspace with the curvature parameter, which measures how “curved” the space is (Figure 2D).

Original: We see that the optimal curvature is about -1, which is the curvature of a standard Poincar´e  ball.
Revised: We see that the optimal curvature is about -1, which is the curvature of a standard Poincar´e [Table] ball.

Original: Additionally, if we gradually change the curvature to 0, the space would be “less curved” and more similar to the Euclidean space (more “ﬂat”).
Revised: Additionally, if we gradually change the curvature to 0, the space would be “less curved” and more similar to the Euclidean space (more “ﬂat”).

Original: Consequently, the Poincar´e scores converge to the Euclidean scores.
Revised: Consequently, the Poincar´e scores converge to the Euclidean scores.

Original: When the curvature is 0, we recover the Euclidean probe.
Revised: When the curvature is 0, we recover the Euclidean probe.

Original: Visualization of Syntax Trees
Revised: Visualization of Syntax Trees

Original: To illustrate the structural differences between Euclidean and
Revised: To illustrate the structural differences between Euclidean and

Original: Poincar´e syntactic subspaces, we visualize the recovered dependency trees in Figure 4.
Revised: Poincar´e syntactic subspaces, we visualize the recovered dependency trees in Figure 4.

Original: To simultaneously visualize edges and tree depth, we jointly train the two probing objectives in equationsand 5 for the same probe.
Revised: To simultaneously visualize edges and tree depth, we jointly train the two probing objectives in equationsand 5 for the same probe.

Original: PCA projection 3 is then used to visualize the syntax trees, similar to Reif et al.
Revised: PCA projection 3 is then used to visualize the syntax trees, similar to Reif et al.

Original: As is shown in Figure 4, compared with the Euclidean probe, the Poincar´e probe shows a more organized word hierarchy: the root word day takes the position at the origin and is surrounded by other words according to tree depth.
Revised: As is shown in Figure 4, compared with the Euclidean probe, the Poincar ´ e probe shows a more organized word hierarchy: the root word day takes the position at the origin and is surrounded by other words according to tree depth.

Original: We further note that embeddings of ELM O [Table] [Table]
Revised: We further note that embeddings of ELM Ocontains no syntactic information and the corresponding tree looks meaningless.

Original: 4. S UMMARY ON S YNTACTIC P ROBE
Revised: 4. S UMMARY ON S YNTACTIC P ROBE

Original: The manifold of BERT embeddings could be complicated and exhibit special geometric properties.
Revised: The manifold of BERT embeddings could be complicated and exhibit special geometric properties.

Original: Our probe essentially serves as a well-deﬁned, differentiable, surjective function that maps the BERT embedding space to a low dimensional, moderately curved Poincar´e ball (rather than a Euclidean space), and consequently leads to better reconstruction of syntax trees.
Revised: Our probe essentially serves as a well-deﬁned, differentiable, surjective function that maps the BERT embedding space to a low dimensional, moderately curved Poincar ´ e ball (rather than a Euclidean space), and consequently leads to better reconstruction of syntax trees.

Original: This indicates that the manifold of BERT syntax may be geometrically more similar to the Poincar ´ e ball than a Euclidean space.
Revised: This indicates that the manifold of BERT syntax may be geometrically more similar to the Poincar ´ e ball than a Euclidean space.

Original: And of course, we cannot conclude that the syntactic subspaces are indeed hyperbolic.
Revised: And of course, we cannot conclude that the syntactic subspaces are indeed hyperbolic.

Original: Rather than being conclusive, we aim to explore alternative models for BERT syntax and reveal the underlying geometric structures.
Revised: Rather than being conclusive, we aim to explore alternative models for BERT syntax and reveal the underlying geometric structures.

Original: Such geometric properties of BERT are still far from well-understood and there are still many open problems worth studying.
Revised: Such geometric properties of BERT are still far from well-understood and there are still many open problems worth studying.

Original:  P ROBING S ENTIMENT
Revised:  P ROBING S ENTIMENT

Original: Having validated the effectiveness of reconstructing syntax trees of our Poincar´e probe, in this section, we generalize the probe to semantic hierarchies and focus on sentiment.
Revised: Having validated the effectiveness of reconstructing syntax trees of our Poincar ´ e probe, in this section, we generalize the probe to semantic hierarchies and focus on sentiment.

Original: We ﬁrst recover a
Revised: We ﬁrst recover a

Original: Poincar´e sentiment subspace with two meta positive and negative embeddings taking the topmost hierarchy and identify word localizations in this subspace.
Revised: Poincar ´ e sentiment subspace with two meta positive and negative embeddings taking the topmost hierarchy and identify word localizations in this subspace.

Original: We highlight that our probe reveals that
Revised: We highlight that our probe reveals that

Original: BERT encodes ﬁne-grained sentiments (positive, negative and neutral ) for each word , even though the probe is trained with sentence-level binary labels .
Revised: BERT encodes ﬁne-grained sentiments (positive, negative and neutral ) for each word , even though the probe is trained with sentence-level binary labels .

Original: To further examine how the context of a sentence may affect its word embeddings, we perform a qualitative lexically-controlled contextualization, i.e., to change the sentiment of a sentence by carefully changing the word choice according to common linguistic rules, and visualize how the localization of embeddings changes accordingly.
Revised: To further examine how the context of a sentence may affect its word embeddings, we perform a qualitative lexically-controlled contextualization, i.e., to change the sentiment of a sentence by carefully changing the word choice according to common linguistic rules, and visualize how the localization of embeddings changes accordingly.

Original: We use the same probe architecture described in Section 3 to construct the sentiment subspace.
Revised: We use the same probe architecture described in Section 3 to construct the sentiment subspace.

Original: Again, a probe is parameterized by two matrices P and Q that project a BERT embedding into a Poincar´e ball.
Revised: Again, a probe is parameterized by two matrices P and Q that project a BERT embedding into a Poincar´e ball.

Original: We adopt sentiment labels for sentences as our supervision and use the Movie
Revised: We adopt sentiment labels for sentences as our supervision and use the Movie

Original: Review dataset (Pang & Lee, 2005) with simple binary labels (positive and negative).
Revised: Review dataset (Pang & Lee, 2005) with simple binary labels (positive and negative).

Original: Details of this dataset are in Appendix B. Given a sentence with t words w 1: t , we project its BERT embeddings according to equation 2 and 3 and obtain q i ∈ D k .
Revised: Details of this dataset are in Appendix B. Given a sentence with t words w 1: t , we project its BERT embeddings according to equation 2 and 3 and obtain q i ∈ D k .

Original: To interpret the classiﬁcation procedure in terms of vector geometry, we set two trainable meta representations for positive and negative labels as c pos , c neg ∈ D k .
Revised: To interpret the classiﬁcation procedure in terms of vector geometry, we set two trainable meta representations for positive and negative labels as c pos , c neg ∈ D k .

Original: The logits for the two classes l pos , l neg are obtained by summing over the Poincar´e 
Revised: The logits for the two classes l pos , l neg are obtained by summing over the Poincar´e it is empirically effective for visualization.

Revised: As there is no perfect analogy of PCA in hyperbolic spaces (Pennec et al., 2018), we leave the investigation of dimension reduction for hyperbolic spaces to future work.

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: distance between each word and the opposite meta embeddings:
Revised: distance between each word and the opposite meta embeddings:

Original: [Equation]
Revised: [Equation]

Original: Since we know that the two classes are contrary to each other, we can manually assign ﬁxed positions for the two meta embeddings.
Revised: Since we know that the two classes are contrary to each other, we can manually assign ﬁxed positions for the two meta embeddings.

Original: For the Euclidean meta embeddings, we use c pos =
Revised: For the Euclidean meta embeddings, we use c pos =

Original: (1 / √ k ) ·and c neg = − c pos in the experiments, where k is the space dimension.
Revised: (1 / √ k ) ·and c neg = − c pos in the experiments, where k is the space dimension.

Original: For the Poincar ´ e meta embeddings, c pos = exp 

Revised: For the Poincar ´ e meta embeddings, c pos = exp

Original: ((1 / √ k ) · 1 ) and c neg = − c pos .
Revised: ((1 / √ k ) · 1 ) and c neg = − c pos .

Original: For training, we use cross-entropy after a Softmax as the loss function.
Revised: For training, we use cross-entropy after a Softmax as the loss function.

Original: An analog method is used for
Revised: An analog method is used for

Original: Euclidean Probes.
Revised: Euclidean Probes.

Original: We use RiemannianAdam (B ´ ecigneul & Ganea, 2019) for all trainable parameters in the Poincar ´ e ball, notably the two meta embeddings, and vanilla Adam for Euclidean parameters.
Revised: We use RiemannianAdam (B ´ ecigneul & Ganea, 2019) for all trainable parameters in the Poincar´e ball, notably the two meta embeddings, and vanilla Adam for Euclidean parameters.

Original: 5. W ORD P OLARITIES B ASED ON G EOMETRIC D ISTANCES
Revised: 5. W ORD P OLARITIES B ASED ON G EOMETRIC D ISTANCES

Original: Table 2 reports the classiﬁcation accuracy.
Revised: Table 2 reports the classiﬁcation accuracy.

Original: Firstly, both probes give better accuracy than a baseline
Revised: Firstly, both probes give better accuracy than a baseline

Original: BiLSTM classiﬁer, meaning that there exists rich sentiment information in BERT embeddings.
Revised: BiLSTM classiﬁer, meaning that there exists rich sentiment information in BERT embeddings.

Original: We highlight that when ﬁxing the class meta representations, the Euclidean probe receives a large loss of performance, while the Poincar´e probe can even perform better.
Revised: We highlight that when ﬁxing the class meta representations, the Euclidean probe receives a large loss of performance, while the Poincar´e probe can even perform better.

Original: This observation strongly suggests that the sentiment information may be encoded in some special geometric way.
Revised: This observation strongly suggests that the sentiment information may be encoded in some special geometric way.

Original: We also report the top sentiment words ranked by the distance gap between two meta-embeddings in Table 3 and see that the words recovered by the Poincar ´ e probe align better with human intuition (colored words).
Revised: We also report the top sentiment words ranked by the distance gap between two meta-embeddings in Table 3 and see that the words recovered by the Poincar´e probe align better with human intuition (colored words).

Original: Classiﬁcation results for each BERT layer and more comprehensive word list are in Appendix B.
Revised: Classiﬁcation results for each BERT layer and more comprehensive word list are in Appendix B.

Original: 5. V ISUALIZATION AND L EXICALLY - CONTROLLED C ONTEXTUALIZATION
Revised: 5. V ISUALIZATION AND L EXICALLY - CONTROLLED C ONTEXTUALIZATION

Original: Figure 5 illustrates how sentences are embedded in the two spaces.
Revised: Figure 5 illustrates how sentences are embedded in the two spaces.

Original: We see that: (a) both probes distinguish very ﬁne-grained word sentiment, as one can infer if each word is positive, negative,  or neutral , even if the probes are trained on sentence-level binary labels;(b) the Poincar´e probe separates two meta-embeddings more clearly than the Euclidean probe and gives more compact embeddings.
Revised: We see that: (a) both probes distinguish very ﬁne-grained word sentiment, as one can infer if each word is positive, negative, [Figure] [Figure] or neutral , even if the probes are trained on sentence-level binary labels;

Revised: (b) the Poincar ´ e probe separates two meta-embeddings more clearly than the Euclidean probe and gives more compact embeddings.

Original: We emphasize that the observation in Figure 5 represents a general pattern, rather than being a special case.
Revised: We emphasize that the observation in Figure 5 represents a general pattern, rather than being a special case.

Original: More visualization can be found in Appendix B.
Revised: More visualization can be found in Appendix B.

Original: To see how different contextualization may change the localization of embeddings, we carefully change the input words to control the sentiment.
Revised: To see how different contextualization may change the localization of embeddings, we carefully change the input words to control the sentiment.

Original: As is shown in Figure 6, we see that: (a) sentiment affects localization: stronger sentiments would result in closer distances to the metaembeddings (Subﬁgure A v.s. B); (b) contextualization affects localization: when the sentence sentiment changes to negative (Subﬁgure A v.s. D), all words will be more close to the negative embedding, even when such change is induced by simple negation (Subﬁgure A v.s. C).
Revised: As is shown in Figure 6, we see that: (a) sentiment affects localization: stronger sentiments would result in closer distances to the metaembeddings (Subﬁgure A v.s. B); (b) contextualization affects localization: when the sentence sentiment changes to negative (Subﬁgure A v.s. D), all words will be more close to the negative embedding, even when such change is induced by simple negation (Subﬁgure A v.s. C).

Original: We further study more complicated cases to explore the limit of BERT embeddings (Figure 7) and ﬁnd out: (a) BERT fails at double negation (Subﬁgure B), which shows that the logical reasoning[Figure] [Figure] behind double negation may be challenging for BERT; (b) BERT gives reasonable localization for an ambiguous sentence (Subﬁgure C) as most words do not signiﬁcantly closer to one meta embeddings (dashed lines); (c) BERT gives correct localization for a sentence with satire (Subﬁgure
Revised: We further study more complicated cases to explore the limit of BERT embeddings (Figure 7) and ﬁnd out: (a) BERT fails at double negation (Subﬁgure B), which shows that the logical reasoning behind double negation may be challenging for BERT; (b) BERT gives reasonable localization for an ambiguous sentence (Subﬁgure C) as most words do not signiﬁcantly closer to one meta embeddings (dashed lines); (c) BERT gives correct localization for a sentence with satire (Subﬁgure

Original: D).
Revised: D).

Original: More cases can be found in Appendix B. We further encourage the reader to run the Jupyter
Revised: More cases can be found in Appendix B. We further encourage the reader to run the Jupyter

Original: Notebook in supplementary materials to discover more visualization results.
Revised: Notebook in supplementary materials to discover more visualization results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present Poincar ´ e probes that recover hyperbolic subspaces for hierarchical information encoded in BERT.
Revised: In this paper, we present Poincar ´ e probes that recover hyperbolic subspaces for hierarchical information encoded in BERT.

Original: With the syntactic probe, we show the possibility that BERT may encode syntax geometrically different from the Euclidean space and the hyperbolic space could be a good alternative.
Revised: With the syntactic probe, we show the possibility that BERT may encode syntax geometrically different from the Euclidean space and the hyperbolic space could be a good alternative.

Original: With the sentiment probe, we further reveal the geometry of BERT embeddings by studying their localization with different contextualization.
Revised: With the sentiment probe, we further reveal the geometry of BERT embeddings by studying their localization with different contextualization.

Original: Our exploration with the Poincar´e probes posit new possibilities about the geometry of BERT embeddings with detailed discussions and extensive visualizations. 
Revised: Our exploration with the Poincar´e probes brings up new possibilities about the geometry of BERT embeddings with detailed discussions and extensive visualizations. A CKNOWLEDGMENTS We thank all the reviewers for their valuable suggestions. This work was supported by Alibaba

Revised: Group through Alibaba Research Intern Program.

Original: A PPENDIX A A DDITIONAL R ESULTS ON P ROBING S YNTAX
Revised: A PPENDIX A A DDITIONAL R ESULTS ON P ROBING S YNTAX

Original: Table 4 shows the score results of training Euclidean and Poincar´e probes for distance and depth tasks simultaneously.
Revised: Table 4 shows the score results of training Euclidean and Poincar´e probes for distance and depth tasks simultaneously.

Original: More plots for longer sentences and deeper trees in the dev set are given (Figure 8 9 10) to illustrate the syntax subspaces (described in Section 3).
Revised: More plots for longer sentences and deeper trees in the dev set are given (Figure 8 9 10) to illustrate the syntax subspaces (described in Section 3).

Original: (B) in these ﬁgures is produced by the 768 d Euclidean Probe, and (C) (D) (E) are produced by the 64 d Poincar´e probe.
Revised: (B) in these ﬁgures is produced by the 768 d Euclidean Probe, and (C) (D) (E) are produced by the 64 d Poincar´e probe.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A.
Revised: A.

Original: ASTRONGER PROBE USING LOCAL SYNTACTIC INFORMATION
Revised: A

Original: Instead of apply a simple linear map as probes, we explore to make a stronger probe which may use local syntactic information of sentences.
Revised: Instead of apply a simple linear map as probes, we explore to make a stronger probe which may use local syntactic information of sentences.

Original: GRU is adopted to project the distributed representation of each word to the syntactical subspaces.
Revised: GRU is adopted to project the distributed representation of each word to the syntactical subspaces.

Original: For Euclidean probe, hidden state of each word in GRU is
Revised: For Euclidean probe, hidden state of each word in GRU is

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: used for encoding syntax, i.e. , q i =
Revised: used for encoding syntax, i.e. , q i =

Original: GRU ( h i ) .
Revised: GRU ( h i ) .

Original: As for Poincar´e probe, the equations are:
Revised: As for Poincar´e probe, the equations are:

Original: [Equation]
Revised: [Equation]

Original: Table 5 reports the scores, which are generally higher than probes based upon linear map.
Revised: Table 5 reports the scores, which are generally higher than probes based upon linear map.

Original: We note both of Poincar´e probe and Euclidean probe obtain much higher scores on ELM O 0, which indicates that these two probes can learn syntax trees, rather than probe from deep models.
Revised: We note both of Poincar´e probe and Euclidean probe obtain much higher scores on ELM O 0, which indicates that these two probes can learn syntax trees, rather than probe from deep models.

Original: Nevertheless, this may suggest the capability of RNN-based models for NLP.
Revised: Nevertheless, this may suggest the capability of RNN-based models for NLP.

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL R ESULTS ON P ROBING S ENTIMENT
Revised: B A DDITIONAL R ESULTS ON P ROBING S ENTIMENT

Original: Movie Review 4 is a balanced sentiment analysis dataset with the same number of positive and negative sentences.
Revised: Movie Review 4 is a balanced sentiment analysis dataset with the same number of positive and negative sentences.

Original: Since there is no ofﬁcial split of this dataset, we randomly split 10% as dev and test set separately.
Revised: Since there is no ofﬁcial split of this dataset, we randomly split 10% as dev and test set separately.

Original: The statistics can be found in Table [Table]
Revised: The statistics can be found in Table 6.

Original: Review

Original: [Table]
Revised: [Table]

Original: Figure 12 reports classiﬁcation accuracy for the probes on each BERT layer.
Revised: Figure 12 reports classiﬁcation accuracy for the probes on each BERT layer.

Original: The BiLSTM baseline utilizes 300 d pretrained G LO V E word embeddings (Pennington et al., 2014) and 300 d hidden states.
Revised: The BiLSTM baseline utilizes 300 d pretrained G LO V E word embeddings ( ? ) and 300 d hidden states.

Original: Table 8 is the extended version of Table 3.
Revised: Table 8 is the extended version of Table 3.

Original: Since some neutral words can be equally close to the two meta embeddings, we reports top sentiment words ranked by the distance gap between two meta-embeddings.
Revised: Since some neutral words can be equally close to the two meta embeddings, we reports top sentiment words ranked by the distance gap between two meta-embeddings.

Original: All subwords (begin with “##”) and numbers are ignored.
Revised: All subwords (begin with “##”) and numbers are ignored.

Original: More plots for sentences in the Movie Review dev set are given (Figure 13 14 15 16 17).
Revised: More plots for sentences in the Movie Review dev set are given (Figure 13 14 15 16 17).

Original: Some neutral words with dashed lines are omitted for clarity.
Revised: Some neutral words with dashed lines are omitted for clarity.

Original: We note that in Figure 13, the positive part is more distinctly produced by Poincar´e probe than Euclidean probe.
Revised: We note that in Figure 13, the positive part is more distinctly produced by Poincar´e probe than Euclidean probe.

Original: In Figure 14, Poincar´e probe correctly realizes ” but ” is turing to positive, but Euclidean Probe notes it as slightly negative, which is used in most cases.
Revised: In Figure 14, Poincar´e probe correctly realizes ” but ” is turing to positive, but Euclidean Probe notes it as slightly negative, which is used in most cases.

Original: A similar pattern can be found in Figure 15.
Revised: A similar pattern can be found in Figure 15.

Original: Figure 16 demonstrates the case that neither Poincar´e probe nor Euclidean probe can correctly classify the sentence, as they may be disturbed by the name entity.
Revised: Figure 16 demonstrates the case that neither Poincar´e probe nor Euclidean probe can correctly classify the sentence, as they may be disturbed by the name entity.

Original: C C LOSED - FORM F ORMULAS OF M ¨ OBIUS O PERATIONS
Revised: C C LOSED - FORM F ORMULAS OF M ¨ OBIUS O PERATIONS

Original: We restate the deﬁnitions of fundamental mathematical operations for the generalized Poincar´e ball model.
Revised: We restate the deﬁnitions of fundamental mathematical operations for the generalized Poincar´e ball model.

Original: We refer readers to Ganea et al.
Revised: We refer readers to Ganea et al.

Original: (2018) for more details
Revised: (2018) for more details

Original: [Table] [Table]
Revised: [Table] [Table]

Original: foolish

Original: M¨obius Addition
Revised: M¨obius Addition

Original: The M¨obius addition for x ,
Revised: The M¨obius addition for x ,

Original: y ∈ D nc is deﬁned as:
Revised: y ∈ D nc is deﬁned as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: M¨obius Matrix-vector Multiplication
Revised: M¨obius Matrix-vector Multiplication

Original: For a linear map M : R n → R m and ∀ x ∈ D nc , if
Revised: For a linear map M : R n → R m and ∀ x ∈ D nc , if

Original: Mx (cid:54) = 0 , then the M¨obius matrix-vector multiplication is deﬁned as: 
Revised: Mx (cid:54) = 0 , then the M¨obius matrix-vector multiplication is deﬁned as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and M ⊗ c x = 

Revised: and M ⊗ c x =

Original: if Mx = 0 .
Revised: if Mx = 0 .

Original: Exponential and Logarithmic Maps
Revised: Exponential and Logarithmic Maps

Original: Let T x D nc denote the tangent space of D nc at x .
Revised: Let T x D nc denote the tangent space of D nc at x .

Original: The exponential map exp c x ( · ) : T x D nc → D nc for v (cid:54) = 0 is deﬁned as:
Revised: The exponential map exp c x ( · ) : T x D nc → D nc for v (cid:54) = 0 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: As the inverse of exp c x ( · ) , the logarithmic map log c x ( · ) : D nc → T x D nc for y (cid:54) = x is deﬁned as:
Revised: As the inverse of exp c x ( · ) , the logarithmic map log c x ( · ) : D nc → T x D nc for y (cid:54) = x is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: D A DDITIONAL R ESULTS ON E UCLIDEAN P ROBE V ARIATIONS
Revised: D A DDITIONAL R ESULTS ON E UCLIDEAN P ROBE V ARIATIONS

Original: We extend Euclidean probes to two linear transforms with non-linearity in between, which would make it more fair comparing to Poincar´e Probe.
Revised: We extend Euclidean probes to two linear transforms with non-linearity in between, which would make it more fair comparing to Poincar´e Probe.

Original: The additional results are reported in Table 9.
Revised: The additional results are reported in Table 9.

Original: [Table]
Revised: [Table]

Original: E A DDITIONAL R ESULTS ON C URVATURE
Revised: E A DDITIONAL R ESULTS ON C URVATURE

Original: To visualize the embedded trees directly in 2 d hyperbolic spaces
Revised: To visualize the embedded trees directly in 2 d hyperbolic spaces

Original: [Table]
Revised: [Table]

Original: we train 2 d probes with different curvatures for both distance and depth tasks simultaneously on BERT BASE 7.
Revised: we train 2 d probes with different curvatures for both distance and depth tasks simultaneously on BERT BASE 7.

Original: The scores are reported in Table 10.
Revised: The scores are reported in Table 10.

Original: The circles in (B), (C), (D) of Figure 18, 19, 20 show the boundary of the
Revised: The circles in (B), (C), (D) of Figure 18, 19, 20 show the boundary of the

Original: Poincar ´ e models.
Revised: Poincar ´ e models.

Original: [Table]
Revised: [Table]

Original: Is this a case where private markets are approving of Washington ’s bashing of Wall Street ? . . . Is this a case where private markets are approving of Washington ’s bashing of Wall Street ? . . .
Revised: Is this a case where private markets are approving of Washington ’s bashing of Wall Street ? . . . Is this a case where private markets are approving of Washington ’s bashing of Wall Street ? . . .

Original: Is this a case where private markets are approving of Washington ’s bashing of Wall Street ? . . . .
Revised: Is this a case where private markets are approving of Washington ’s bashing of Wall Street ? . . . .

Original: [Table]
Revised: [Table]

Original: Baseball 
Revised: Baseball ,

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Francisco Giants in the World Series last weekend . . . .
Revised: Giants in the World Series last weekend . . . .

Original: Indeed , the Dow Jones Transportation Average plunged 102.06 points , its second-worst drop in history .
Revised: Indeed , the Dow Jones Transportation Average plunged 102.06 points , its second-worst drop in history .

Original: Indeed , the Dow Jones Transportation Average plunged 102.06 points , its second-worst drop in history . Indeed , the Dow Jones Transportation Average plunged 102.06 points , its second-worst drop in history . .
Revised: Indeed , the Dow Jones Transportation Average plunged 102.06 points , its second-worst drop in history . Indeed , the Dow Jones Transportation Average plunged 102.06 points , its second-worst drop in history . .

Original: U.S. Banknote said it is in negotiations to sell “ certain facilities , ” which it did n’t name , to a third party , and it needs the extension to try to reach a deﬁnitive agreement on the sale . .
Revised: U.S. Banknote said it is in negotiations to sell “ certain facilities , ” which it did n’t name , to a third party , and it needs the extension to try to reach a deﬁnitive agreement on the sale . .

Original: U.S. Banknote said it is in negotiations to sell “ certain facilities , ” which it did n’t name , to a third party , and it needs the extension to try to reach a deﬁnitive agreement on the sale . .U.S Banknote said it is in negotiations [Table]
Revised: U.S. Banknote said it is in negotiations to sell “ certain facilities , ” which it did n’t name , to a third party , and it needs the extension to try to reach a deﬁnitive agreement on the sale . .

Original: [Table]

Original: [Table]

Original: it

Original: [Table]
Revised: [Table]

Original: n’t

Original: [Table][Table] party , [Table]
Revised: [Table]

Original: it needs

Original: [Table]
Revised: [Table]

Original: extension

Original: [Table]
Revised: [Table]

Original: try

Original: [Table]
Revised: [Table]

Original: deﬁnitive agreement

Original: [Table]
Revised: [Table]

Original: sale . . . . Isao Ushikubo , general manager of the investment research department at Toyo Trust + Banking Co. , also was optimistic . . Isao Ushikubo , general manager of the investment research department at Toyo Trust + Banking Co. , also was optimistic . .
Revised: Isao Ushikubo , general manager of the investment research department at Toyo Trust + Banking Co. , also was optimistic . . Isao Ushikubo , general manager of the investment research department at Toyo Trust + Banking Co. , also was optimistic . .

Original: Isao Ushikubo , general manager of the investment research department at Toyo Trust + Banking Co. , also was optimistic . . .
Revised: Isao Ushikubo , general manager of the investment research department at Toyo Trust + Banking Co. , also was optimistic . . .

Original: (5) “ In the U.S. market , the recognition of the Manpower name is inﬁnitely stronger than Blue Arrow , ” Mr. Fromstein said . .
Revised: (5) “ In the U.S. market , the recognition of the Manpower name is inﬁnitely stronger than Blue Arrow , ” Mr. Fromstein said . .

Original: “ In the U.S. market , the recognition of the Manpower name is inﬁnitely stronger than Blue Arrow , ” Mr. Fromstein said . . “ In the U.S. market , the recognition of the Manpower name is inﬁnitely stronger than Blue Arrow , ” Mr. Fromstein said . .
Revised: “ In the U.S. market , the recognition of the Manpower name is inﬁnitely stronger than Blue Arrow , ” Mr. Fromstein said . . “ In the U.S. market , the recognition of the Manpower name is inﬁnitely stronger than Blue Arrow , ” Mr. Fromstein said . .

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: For the moment , at least , euphoria has replaced anxiety on Wall Street . For the moment , at least , euphoria has replaced anxiety on Wall Street .
Revised: For the moment , at least , euphoria has replaced anxiety on Wall Street . For the moment , at least , euphoria has replaced anxiety on Wall Street .

Original: For the moment , at least , euphoria has replaced anxiety on Wall Street .
Revised: For the moment , at least , euphoria has replaced anxiety on Wall Street .

Original: (8) “ We went down 3/4 point in 10 minutes right before lunch , then after lunch we went up 3/4 point in 12 minutes , ” he said . .
Revised: (8) “ We went down 3/4 point in 10 minutes right before lunch , then after lunch we went up 3/4 point in 12 minutes , ” he said . .

Original: “ We went down 3/4 point in 10 minutes right before lunch , then after lunch we went up 3/4 point in 12 minutes , ” he said . . “ We went down 3/4 point in 10 minutes right before lunch , then after lunch we went up 3/4 point in 12 minutes , ” he said . .
Revised: “ We went down 3/4 point in 10 minutes right before lunch , then after lunch we went up 3/4 point in 12 minutes , ” he said . . “ We went down 3/4 point in 10 minutes right before lunch , then after lunch we went up 3/4 point in 12 minutes , ” he said . .

Original: The percentage rates are calculated on a 360-day year , while the coupon-equivalent yield is based on a 365-day year . .
Revised: The percentage rates are calculated on a [Table] day year , [Table]

Revised: yield

Original: The percentage rates are calculated on a 360-day year , while the coupon-equivalent yield is based on a 365-day year . .
Revised: [Table] based on a [Table] day year . .

Revised: The

Revised: [Table]

Revised: rates

Original:  The percentage rates are calculated on a 360-day year , while the coupon-equivalent yield is based on a 365-day year . . .
Revised: [Table] day year , [Table] The percentage rates are calculated on a 360-day year , while the coupon-equivalent yield is based on a 365-day year . . .

Original: Mr. Farley followed a similar pattern when he acquired Northwest Industries Inc. and then sold much of its assets . .
Revised: Mr. Farley followed a similar pattern when he acquired Northwest Industries Inc. and then sold much of its assets . .

Original: Mr. Farley followed a similar pattern when he acquired Northwest Industries Inc. and then sold much of its assets . . Mr. Farley followed a similar pattern when he acquired Northwest Industries Inc. and then sold much of its assets . .
Revised: Mr. Farley followed a similar pattern when he acquired Northwest Industries Inc. and then sold much of its assets . . Mr. Farley followed a similar pattern when he acquired Northwest Industries Inc. and then sold much of its assets . .

Original: Among Big Board specialists , the cry was “ Pull your offers ” – meaning that specialists soon expected to get higher prices for their shares . . Among Big Board specialists , the cry was “ Pull your offers ” – meaning that specialists soon expected to get higher prices for their shares . .
Revised: Among Big Board specialists , the cry was “ Pull your offers ” – meaning that specialists soon expected to get higher prices for their shares . . Among Big Board specialists , the cry was “ Pull your offers ” – meaning that specialists soon expected to get higher prices for their shares . .

Original: Among Big Board specialists , the cry was “ Pull your offers ” – meaning that specialists soon expected to get higher prices for their shares . . . . . .
Revised: Among Big Board specialists , the cry was “ Pull your offers ” – meaning that specialists soon expected to get higher prices for their shares . . . . . .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]