  Linear embedding transformation has been shown to be effective for zero-shot
cross-lingual transfer tasks and achieve surprisingly promising results.
However, cross-lingual embedding space mapping is usually studied in static
word-level embeddings, where a space transformation is derived by aligning
representations of translation pairs that are referred from dictionaries. We
move further from this line and investigate a contextual embedding alignment
approach which is sense-level and dictionary-free. To enhance the quality of
the mapping, we also provide a deep view of properties of contextual
embeddings, i.e., anisotropy problem and its solution. Experiments on zero-shot
dependency parsing through the concept-shared space built by our embedding
transformation substantially outperform state-of-the-art methods using
multilingual embeddings.
