Typology Guided Multilingual Position Representations: Case on Dependency Parsing (2023)

Abstract: Recent multilingual models benefit from strong unified semantic representation models. However, due to conflict linguistic regularities, ignoring language-specific features during multilingual learning may suffer from negative transfer. In this work, we analyze the relationbetween a language’s position space and its typological characterization, and suggest deploying different position spaces for different languages. We develop a position generation network which combines prior knowledge from typology features and existing position vectors. Experiments on the multilingual dependency parsing task show that the learned position vectors exhibit meaningful hidden structures, and they can help achieving the best multilingual parsing results.

## introduction with the recent progress on multilingual text representations, there has been a growing interest in developing unified models for nlp tasks crossing different languages @xcite @xcite . for high-resource languages, a unified multilingual model is faster to train and easier to deploy than a bunch of independent monolingual models. for low(zero)-resource languages, a multilingual model may help building positive knowledge transfer among languages. words and their positions in the text are two main features of any language's sentences. for words, various multilingual pre-trained models @xcite and alignment algorithms @xcite have been devoted to unifying lexical semantic spaces among languages. for positions, however, there are much less study on their roles in joint multilingual learning: models simply adopt one universal position representation for all languages. since word positions describe word orders, a single position space implies all languages are compiled under the same word order system, which is not true according to linguistic prior. for example, adjectives are usually placed before nouns in english, while in french they are almost after nouns. such conflicts of linguistic regularities may break the effectiveness of word position features in multilingual learning (especially for those tasks sensitive to word order (e.g., syntactic and semantic parsing @xcite )). in this paper, we study the connection between a language's position space and its typological characterization (especially, word order characterization). by jointly learning position spaces with a syntactic parsing task, we first have two findings. we next develop methods to construct multilingual position representations. options may include attaching language ids to the existing universal position space @xcite and learning position representations from scratch @xcite . one main concern on those approaches is on handling unseen languages: if a language doesn't appear in the training set, its position representations are totally unknown. our key technical contribution is a generation network for positions. it explicitly takes word typological features of a language as input and outputs a set of position vectors for that language. for unseen languages, we are free to obtain their position vectors through prior on their typology. during the generation process, we take the universal position representations as basis vectors for each language's position space, which makes the learned vectors still carry the prior of "representing a position in texts". under this setting, we are able to examine the shift of languages' position spaces with a unified coordinate system. we take multilingual dependency parsing as our demonstrating task. the parser is trained on 13 languages from the universal dependencies treebanks, and is tested with both languages present (13) and absent (30) in the training set. the results show that, with the typological guided position vectors, the parser is able to achieve both significant improvements on seen (+4.1 las) and unseen (+1.2 las) languages compared with using universal position representations. ## preliminary our multilingual models take transformer @xcite as basic building blocks. there are two types of position spaces, absolute and relative. ## position spaces for multilingual learning existing mutlilingual models use a universal position space for all languages. it is questionable that whether one position space is enough to handle languages with different linguistic constraints. in order to inspect relations between position representations and typological features, we experiment a multilingual parser with language-specific position vectors. for each language, the model assigns a set of learnable vectors for each position (absolute or relative), and the position vectors are jointly learned with the parser. first, we examine whether the learned position vectors carry information about word order. taking the order of subject(s), verb(v) and object(o) as an example, we merge datasets of english_en (svo) and hindi_hi (sov), and train a binary probing classifier to discriminate two word orders. the classifier is a 2-layer mlp taking the mean pooling of 30 25 20 15 10 5 0 accuracy a) svo sov sv ru fi zh hi ko tr ja he ar eu 30 25 20 15 10 5 0 accuracy b) an na figure 2: the probing accuracy of different language groups. a) svo or sov, b) na or an the parser's final layer hidden vectors as input. following the general probing workflow, we may use testing accuracies of the classifier to assert whether word order information is encoded. however, a high probing accuracy is not trustworthy here because the vocabulary overlapping of two languages is usually small, and the probing classifier is able to achieve high accuracies by simply ignoring the actual word order features and only recognizing differences of the two distinct vocabularies. we adopt a different probing strategy. after training the probing classifier on english and hindi, remaining languages are then divided into two groups, the svo group (chinese_zh, finnish_fi, hebrew_he, italian_it, russian_ru, swedish_sv) and the sov group (basque_eu, japanese_ja, ko-rean_ko, turkish_tr). we replace the position vectors of english with those of the two groups, and investigate the accuracy of svo recognition on english. if position vectors have successfully learned the concept of word order across different languages, we can expected a better probing performances when the replaced vectors are from the same group. figure 2 shows that on english, position vectors from svo languages performs much better than sov languages. the results on nounadjective order (na or an) are similar. second, we can further ask whether the distance between two position spaces reflects the typological distance between two languages. we choose a linguistic distance metric defined by @xcite . for position spaces, we compute the average cosine similarities of two corresponding position vectors. figure 1 shows that the two distances are highly correlated: similar languages have similar position spaces. it suggests that the customized position vectors may be consulted for avoiding negative transfer in multilingual learning. in fact, we perform another substitution experiment directly on the learned parser (figure 1 ). by replacing english position vectors with distant languages (e.g., japanese), the parsing performances drop a lot. therefore if we unify languages with a universal position space, the conflict of language regularities may cause negative transfer. finally, we compare overall multilingual parsing performances when learning with universal position vectors and learning with different position vectors. figure 1 shows that the latter always performs better (+2.8 average las). ## typology-guided position generation analyses above suggest us to apply different position spaces for different languages. however, naively assigning learnable vectors for positions can not generalized to unseen languages. in order to make the multilingual model applicable to languages not appear in the training set, we propose to generate position vectors under explicit guidance of typological features. ## conclusions we studied the role of position spaces in multilingual learning. by comparing a univeral position space and language-specific position spaces, we showed the latter could either handle linguistic constraints of different language efficiently or provide a clear path for positive transfer in multilingual learning. we developped a self-attention based position space generator. we showed that by utilizing typological prior and existing position space prior, the multilingual dependency parser could enjoy positive transfer on both high-resource and zeroresource languages. one future work is to investigate whether the obtained position vectors could help other multilingual and monolingual tasks. it is also interesting to compare the position spaces induced from different multilingual tasks (supervised or unsupervised).