  With the recent success of pre-trained models in NLP, a significant focus was
put on interpreting their representations. One of the most prominent approaches
is structural probing (Hewitt and Manning, 2019), where a linear projection of
language vector space is performed in order to approximate the topology of
linguistic structures. In this work, we decompose this mapping into 1.
isomorphic space rotation; 2. linear scaling that identifies and scales the
most relevant directions. We introduce novel structural tasks to exam our
method's ability to disentangle information hidden in the embeddings. We
experimentally show that our approach can be performed in a multitask setting.
Moreover, the orthogonal constraint identifies embedding subspaces encoding
specific linguistic features and make the probe less vulnerable to
memorization.
