  Recently, a variety of probing tasks are proposed to discover linguistic
properties learned in contextualized word embeddings. Many of these works
implicitly assume these embeddings lay in certain metric spaces, typically the
Euclidean space. This work considers a family of geometrically special spaces,
the hyperbolic spaces, that exhibit better inductive biases for hierarchical
structures and may better reveal linguistic hierarchies encoded in
contextualized representations. We introduce a Poincare probe, a structural
probe projecting these embeddings into a Poincare subspace with explicitly
defined hierarchies. We focus on two probing objectives: (a) dependency trees
where the hierarchy is defined as head-dependent structures; (b) lexical
sentiments where the hierarchy is defined as the polarity of words (positivity
and negativity). We argue that a key desideratum of a probe is its sensitivity
to the existence of linguistic structures. We apply our probes on BERT, a
typical contextualized embedding model. In a syntactic subspace, our probe
better recovers tree structures than Euclidean probes, revealing the
possibility that the geometry of BERT syntax may not necessarily be Euclidean.
In a sentiment subspace, we reveal two possible meta-embeddings for positive
and negative sentiments and show how lexically-controlled contextualization
would change the geometric localization of embeddings. We demonstrate the
findings with our Poincare probe via extensive experiments and visualization.
Our results can be reproduced at https://github.com/FranxYao/PoincareProbe.
