Improving Cross-lingual Transfer through Subtree-aware Word Reordering (2023)

Abstract: Despite the impressive growth of the abilities of multilingual language models, such as XLM-R and mT5, it has been shown that they still face difficulties when tackling typologically-distant languages, particularly in the low-resource setting. One obstacle for effective cross-lingual transfer is variability in word-order patterns. It can be potentially mitigated via source- or target-side word reordering, and numerous approaches to reordering have been proposed. However, they rely on language-specific rules, work on the level of POS tags, or only target the main clause, leaving subordinate clauses intact. To address these limitations, we present a new powerful reordering method, defined in terms of Universal Dependencies, that is able to learn fine-grained word-order patterns conditioned on the syntactic context from a small amount of annotated data and can be applied at all levels of the syntactic tree. We conduct experiments on a diverse set of tasks and show that our method consistently outperforms strong baselines over different language pairs and model architectures. This performance advantage holds true in both zero-shot and few-shot scenarios.

## introduction recent multilingual pre-trained language models (lms), such as mbert @xcite , xlm-roberta @xcite , mbart @xcite , and mt5 @xcite , have shown impressive cross-lingual ability, enabling effective transfer in a wide range of cross-lingual natural language processing tasks. however, even the most advanced llms are not effective when dealing with less-represented languages, as shown by recent studies @xcite @xcite . furthermore, annotating sufficient training data in these languages is not a feasible task, and as a result speakers of underrepresented languages are unable to reap the benefits of modern nlp capabilities @xcite . numerous studies have shown that a key challenge for cross-lingual transfer is the divergence in word order between different languages, which often causes a significant drop in performance @xcite @xcite @xcite @xcite . 2 this is unsurprising, given the complex and interdependent nature of word-order (e.g., verb-final languages tend to have postpositions instead of prepositions and place relative clauses before nominal phrases that they modify, while svo and vso languages prefer prepositions and postposed relative clauses, see dryer 1992) and the way it is coupled with the presentation of novel information in sentences @xcite . this is especially true for the majority of underrepresented languages, which demonstrate distinct word order preferences from english and other well-resourced languages. motivated by this, we present a reordering method applicable to any language pair, which can be efficiently trained even on a small amount of data, is applicable at all levels of the syntactic tree, and is powerful enough to boost the performance of modern multilingual lms. the method, defined in terms of universal dependencies (ud), is based on pairwise constraints regulating the linear order of subtrees that share a common parent, which we term pocs for "pairwise ordering constraints". we estimate these constraints based on the probability that the two subtree labels will appear in one order or the other when their parent has a given label. thus, in terms of ud, we expect, e.g., languages that use pre-nominal adjectival modification to assign a high probability to amods' preceding their headword, while languages with post-nominal adjectival modification are expected to have a high probability for the other direction. the estimated pocs are fed as constraints to an smt solver foot_2 to produce a general reordering algorithm that can affect reordering of all types of syntactic structures. in addition to being effective, pocs are interpretable and provide a detailed characterization of typical word order patterns in different languages, allowing interpretability as to the effect of word order on cross-lingual transfer. we evaluate our method on three cross-lingual tasks -dependency parsing, task-oriented semantic parsing, and relation classification -in the zeroshot setting. such setting is practically useful (see, e.g., @xcite @xcite for successful examples of employing zs learning cross-lingually) and minimizes the risk of introducing confounds into the analysis. we further evaluate our method in the scarcedata scenario on the semantic parsing task. this scenario is more realistic as in many cases it is feasible to annotate small amounts of data in specific languages @xcite . experiments show that our method consistently presents a noticeable performance gain compared to the baselines over different language pairs and model architectures, both in the zero-shot and fewshot scenario. this suggests that despite recent advances, stronger multilingual models still faces difficulties in handling cross-lingual word order divergences, and that reordering algorithms, such as ours, can provide a much needed boost in performance in low-resource languages. additionally, we investigate the relative effectiveness of our reordering algorithm on two types of neural architectures: encoder-decoder (seq2seq) vs. a classification head stacked on top of a pretrained encoder. our findings show that the encoder-decoder architecture underperforms in cross-lingual transfer and benefits more strongly from reordering, suggesting that it may struggle with projecting patterns over word-order divergences. the structure of the paper is as follows: section 2 surveys related work. the proposed approach is introduced in section 3. section 4 de-scribes the setup for our zero-shot and few-shot experiments, the results of which are presented in section 5. section 6 investigates the comparative performance of encoder-based and sequenceto-sequence models, and section 7 concludes the paper. ## related work a major challenge for cross-lingual transfer stems from word-order differences between the source and target language. this challenge has been the subject of many previous works (e.g., @xcite , and numerous approaches to overcome it have been proposed. one of the major approaches of this type is reordering, i.e. rearranging the word order in the source sentences to make them more similar to the target ones or vice versa. early approaches, mainly in phrase-based statistical machine translation, relied on hand-written rules @xcite , while later attempts were made to extract reordering rules automatically using parallel corpora by minimizing the number of crossing wordalignments @xcite . more recent works focusing on reordering relied on statistics of various linguistic properties such as pos-tags @xcite @xcite and syntactic relations (rasooli and @xcite . such statistics can be taken from typological datasets such as wals @xcite or extracted from large corpora @xcite . other works proposed to make architectural changes in the models. thus, @xcite al. (2017a our work is in line with the proposed solutions to source-sentence reordering, namely treebank reordering, which aim to rearrange the word order of source sentences by linearly permuting the nodes of their dependency-parse trees. @xcite and @xcite suggested permuting existing dependency treebanks to make their surface pos-sequence statistics close to those of the target language, in order to improve the performance of delexicalized dependency parsers in the zero-shot scenario. while some improvements were reported, these approaches rely on short pos n-grams and do not capture many important patterns. 4 @xcite proposed a similar method but used a pos-based language model, trained on a target-language corpus, to guide their algorithm. this provided them with the ability to capture more complex statistics, but utilizing black-box learned models renders their method difficult to interpret. rasooli and @xcite proposed a reordering algorithm based on ud, specifically, the dominant dependency direction in the target language, leveraging the rich syntactic information the annotation provides. their method however, leverages only a small part of ud richness, compared to our method. we note that previous work on treebank reordering usually only evaluated their methods on ud parsing, using delexicalized models or simple manually aligned cross-lingual word embeddings, which limited the scope of the analysis. in this paper, we experiment with two additional tasks that are not reducible to syntactic parsing: relation classification and semantic parsing. we further extend previous work by using modern multilingual lms and experimenting with different architectures. ## approach given a sentence @xmath0, s 2 , ..., s n in source language l s , we aim to permute the words in it to mimic the word order of a target language l t . similarly to previous works @xcite , we make the assumption that a contiguous subsequence that forms a constituent in the original sentence should remain a contiguous subsequence after reordering, while the inner order of words in it may change. this prevents subtrees from losing their semantic coherence and is also vital when dealing with tasks such as relation extraction (re), where some of the subequence must stay intact in order for the annotation to remain valid. concretely, instead of permuting the words of sentence s, we permute the subtrees of its ud parse tree, thus keeping the subsequences of s, as 4 @xcite further experimented with manually crafting permutations rules using typological data on pos sequences from wals. this approach is less demanding in terms of data but is more labor intensive and does not lead to better performance. defined by the parse-tree structure, intact. 5 we define a set of language-specific constraintsbased on the notion of pairwise ordering distributions, the tendency of words with specific ud labels to be linearly ordered before words with other specific labels, conditioned on the type of subtree they appear in. to implement a reodering algorithm we use these constraints as input to an smt solver. ## experimental setup we evaluate our reordering algorithm using three tasks -ud parsing, task-oriented semantic parsing, and relation extraction -and over 13 different target languages, with english as the source language. 9 for each task and target language, we compare the performance of a model trained on the vanilla english dataset against that of a model trained on a transformed (reordered) version of the dataset, using the target-language test set in a zero-shot fashion. we explore two settings: standard, where we reorder the english dataset according to the target language pocs and use it for training, and en-semble, where we train our models on both the vanilla and the reordered english datasets. the main motivation for this is that any reordering algorithm is bound to add noise to the data. first, the underlying multilingual lms were trained on the standard word order of english, and feeding them english sentences in an unnatural word order will likely produce sub-optimal representations. secondly, reordering algorithms rely on surface statistics, which, while rich, are a product of statistical estimation and thus imperfect. lastly, the use of hard constrains may not be justified for target languages with highly flexible word-order 10 . the ensemble setting mitigates these issues and improves the "signal to noise ratio" of the approach. we use the vanilla multilingual models and the reordering algorithm by @xcite as baselines. to the best of our knowledge, rasooli and collins proposed the most recent preprocessing reordering algorithm, which also relies on ud annotation. we re-implement the algorithm and use it in the same settings as our approach. lastly, we evaluate our method in the scarce-data setting. we additionally train the models fine-tuned on the vanilla and reordered english datasets on a small number of examples in the target language and record their performance. due to the large amount of experiments required we conduct this experiment using only our method in the context of the semantic-parsing task, which is the most challenging one in our benchmark @xcite , on the mt5 model, and in the ensemble setting. ## results and discussion the results on the various tasks, namely ud parsing, semantic parsing, and relation classification, are presented in tables 1, 2 and 3 respectively. the few-shot experiment results are in in ud parsing, the ensemble setting presents noticeable improvements for the languages that are more typologically distant from english (2.3-4.1 las points and 1.8-3.5 uas points), with the exception of arabic, in which the scores slightly drop. no noticeable effect is observed for structurally closer languages. in the standard setting, a smaller increase in performance is present for most distant languages, with a decrease in performance for closes ones, persian and arabic. this is in agreement with previous work that showed that reordering algorithms are more beneficial when applied to structurallydivergent language pairs @xcite . the ensemble approach, therefore, seems to be essential for a generally applicable reordering algorithm. the algorithm by rasooli and collins (2019) (rc19), in both settings, presents smaller increase in performance for some typologically-distant languages and no noticeable improvements for others, while sometimes harming the results. this suggests that for this task the surface statistics the algorithm uses are not enough and a more fine-grained approach in needed. in the semantic-parsing task, the reordering algorithm presents substantial improvements for all languages but italian in the ensemble setting (2-6.1 increase in exact match), for the roberta based model. noticeably, the gains are achieved not only for typologically-distant languages but also for languages close to english, such as french. in the mtop dataset, the ensemble setting proves bigger gains over the standard for all languages. in multilingual-top, we surprisingly observe the opposite. given that japanese in terms of word order is comparable to hindi and italian, to french, we tend to attribute this result to the peculiarities of the dataset. this, however, merits further analysis. when compared to rc19, the proposed algorithm consistently outperforms it, by an average of about 2 points (in the ensemble setting). for mt5 we observe increase in performance, in the ensemble setting, of 2.5 and 5 points in thai and hindi, respectively. for the other languages, we do not observe a strong impact. we note however, that in french and spanish, there is a slight drop in the score (less then 1 point). when compared to rc19, our method provide larger gains in hindi and thai. in the few-shot scenario, we observe improved performances for all languages and sample sizes. surprisingly, the improvements hold even when training on a large sample size of 500, indicating that the model is not able to easily adapt to the target word-order. lastly, in the relation-classification task, in the ensemble setting we observe an increase in performance for all languages (2.3-10.4 increase in the micro and macro f1 points). in the standard setting, there is a drop in performance for hindi and telugu. when compared to rc19, our algorithm outperforms it in the ensemble setting, by more than 5 points for korean, but only by 0.5 points for russian. in hindi and telugu, the performance of both algorithms is close, and rc19 does perform better in some cases. ## comparison between encoder-with-classifier-head and seq2seq models past work has shown that the architecture is an important predictor of the ability of a given model to generalize over cross-lingual word-order divergences. for example, @xcite showed that models based on self-attention have a better overall cross-lingual transferability to distant languages than those using rnn-based architectures. one of the dominant trends in recent years in nlp has been using the sequence-to-sequence formulation to solve an increasing variety of tasks @xcite . despite that, recent studies @xcite @xcite demonstrated that such models fail at compositional generalization, that is, they do not generalize to structures that were not seen at training time. @xcite showed that other model architectures can prove advantageous over seq2seq architecture in that regard, but their work was limited to english. here, we take the first steps in examining the cross-lingual transfer capabilities of the seq2seq encoder-decoder architecture (s2s) vs. a classification head stacked over an encoder (e+c), focusing on their ability to bridge word-order divergences. ## conclusion we presented a novel pre-processing reordering approach that is defined in terms of universal dependencies. experiments on three tasks and numerous architectures and target languages demonstrate that this method is able to boost the performances of modern multilingual lms both in the zero-shot and few-shot setting. our key contributions include a new method for reordering sentences based on fine-grained word-order statistics, the pairwise ordering distributions, using an smt solver to convert the learned constraints into a linear ordering, and a demonstration of the necessity of combining the reordered dataset with the original one (the ensemble setting) in order to consistently boost performance. our results suggest that despite the recent improvements in multilingual models, they still face difficulties in handling cross-lingual word order divergences, and that reordering algorithms, such as ours, can provide a much needed boost in performance in low-resource languages. this result holds even in the few-shot scenario, when the model is trained on few hundred examples, underscoring the difficulty of models to adapt to varying word orders, as well as the need for more typologically diverse data, additional inductive bias at the training time, or a pre-processing approach such as ours to be more effective. furthermore, our experiments suggest that seq2seq encoder-decoder architectures may suffer from these difficulties to a bigger extent than more traditional modular ones. future work will include, firstly, addressing the limitations of the proposed approach in order to make it less language-pair dependent and reduce the computational and storage overhead, and secondly, leveraging the pocs in order to compute the word-order distance between languages in a rich, rigorous corpus-based way, 14 and to more precisely predict when the reordering algorithm will be beneficial as well as to provide a fine-grained analysis of the connection between word order and cross-lingual performance, in line with @xcite .