Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation (2021)

Abstract: Linear embedding transformation has been shown to be effective for zero-shot cross-lingual transfer tasks and achieve surprisingly promising results. However, cross-lingual embedding space mapping is usually studied in static word-level embeddings, where a space transformation is derived by aligning representations of translation pairs that are referred from dictionaries. We move further from this line and investigate a contextual embedding alignment approach which is sense-level and dictionary-free. To enhance the quality of the mapping, we also provide a deep view of properties of contextual embeddings, i.e., the anisotropy problem and its solution. Experiments on zero-shot dependency parsing through the concept-shared space built by our embedding transformation substantially outperform state-of-the-art methods using multilingual embeddings.

## introduction cross-lingual embedding space alignment @xcite @xcite recently has been attracted a lot of attention because cross-lingual model transfer is effectively facilitated by shared semantic spaces in nlp tasks, e.g., named entity recognition @xcite , part-of-speech tagging @xcite , and dependency parsing @xcite , where dependency paring is scoped out in this paper. compared with the delexicalized parsers @xcite , multilingual word embeddings have been demonstrated to significantly improve the performance of zero-shot dependency parsing by bridging the lexical feature gap @xcite . with the remarkable development of monolingual contextual pre-trained models @xcite @xcite , which dramatically outperform static word embeddings @xcite @xcite in broad nlp applications, increasing number of researchers have started focusing on contextual representation alignment for cross-lingual dependency parsing @xcite . moreover, with the appearance of multilingual pre-trained models, such as multilingual bert (mbert) @xcite , zero-shot dependency parsing becomes easier by utilizing the large vocabulary of the multilingual models @xcite . our approach is most similar to @xcite , which maps a target language space into a source language space through a linear transformation to realize zero-shot transfer in dependency parsing. typically, a transformation is usually derived by word-level embedding alignment, while we explore a sense-level embedding alignment method to map bilingual spaces more precisely, where sense-level representations are split from multi-sense word-level embeddings. furthermore, our mapping approach is dictionary-free which utilizes the silver token pairs from parallel corpora and eliminates the necessity of gold dictionaries. the experimental results of zero-shot dependency parsing demonstrate that two parser evaluation scores (uas and las) of sense-level mapping are always better than of word-level one. moreover, we also notice the anisotropy problem @xcite (defined in section 3.2) in contextual embeddings, which potentially deteriorate the performance of the zero-shot transfer task. we significantly mitigate this drawback by leveraging a prepossessing step, iterative normalization (in) @xcite , which is originally used for improving the performance of static embedding mapping on the bilingual dictionary induction task. zero-shot dependency parsing experiments are conducted on universal dependencies treebank v2.6 (zeman et al., 2020), which shows that our results obtain a substantial gain compared with state-of-the-art methods using multilingual fasttext and mbert foot_0 . ## linear cross-lingual space alignment let denote @xmath0, and y as the word embedding matrix for a source language. for each column of the target embedding matrix x @xmath1, it has one source embedding vector y @xmath2. we aim to derive a linear transformation matrix ŵ used for mapping from the target language space to the source language space. this can be learned by minimizing the frobenius norm: furthermore, @xcite show that the quality of space alignment is successfully improved with the orthogonal restriction, i.e, w t @xmath3. thus, the problem can be solved by procrustes approach @xcite : ## experiment our parser is the deep biaffine model from @xcite where hyperparameters are almost unchanged. the settings of all hyperparameters are listed in appendix b. english is set as the source language and other languages are targets. in our experiments, we select 6 target languages from 4 language families for which we have offthe-shelf monolingual pre-trained bert models (base-size). we train the parsing model only in the english treebank, and directly evaluate zero-shot transfer performance on the target languages. ## conclusion we proposed a linear, dictionary-free and senselevel contextual mapping approach by exploiting parallel corpus which has shown promising results and substantial improvement compared with multilingual fasttext and mbert in the zero-shot dependency parsing task. we also revealed that various degrees of anisotropy hurts the performance of mapping, and introduced iterative normalization to alleviate it by enforcing contextual embeddings to be uniformly distributed, which also has indicated the benefits of isotropy.