Word Reordering for Zero-shot Cross-lingual Structured Prediction (2021)

Abstract: Adapting word order from one language to another is a key problem in cross-lingual structured prediction. Current sentence encoders (e.g., RNN, Transformer with position embeddings) are usually word order sensitive. Even with uniform word form representations (MUSE, mBERT), word order discrepancies may hurt the adaptation of models. In this paper, we build structured prediction models with bag-of-words inputs, and introduce a new reordering module to organizing words following the source language order, which learns task-specific reordering strategies from a general-purpose order predictor model. Experiments on zero-shot cross-lingual dependency parsing, POS tagging, and morphological tagging show that our model can significantly improve target language performances, especially for languages that are distant from the source language.

## introduction extracting linguistic structures from natural language usually relies on high quality human annotations. to handle low resource scenarios, efforts have been devoted to sharing resources among languages and adapting from high resource models. one crucial step of these methods is how to unify input and output spaces over languages. for example, the universal dependency project @xcite constructs a universal output space for crosslingual dependency parsers, and cross-lingual word representation learning algorithms helps aligning word forms of different languages @xcite . beyond word form, word order is another important factor in cross-lingual structured prediction @xcite : it is possible that sentences in two different language have similar parse trees, but their words are organized in different orders (e.g., svo or sov). to share annotations among them, we need to handle word order discrepancies carefully: if a model learned on the source language is tightly coupled with the source language word order, performances on target languages could be hurt as their word order could be incompatible @xcite . on the other side, if one completely drops word order (e.g., bag-ofwords), the source language (and target languages) performances might be poor as order-sensitive features could be essential. trade-offs have been made by using weak word order information (e.g., relative positions instead of absolute positions @xcite ), but we still want to seek better adaptation of word order without sacrificing source language performances. in this work, we integrate new reordering modules to help cross-lingual structured prediction. given a bag-of-words from the target language, the module tries to reorder them to best resemble a source language sentence. the structured prediction part then receives inputs with a more familiar order information. crucially, • the training of the reordering model only requires unlabelled source language data, without parallel corpora or off-the-shelf word alignment tools @xcite (thanks to the universal word forms). we suggest a distillation framework @xcite for learning the reordering module. ## method overview given a sentence @xmath0, w 1 , • • • , w n , e (begin and end are synthetic marks), we can have two views on x, a bag-of-words ω and an order of words o. structured prediction tasks map x to a linguistic structure y (e.g., a parse tree). here, we consider a strict zero-shot cross-lingual setting: given a labelled source language corpus @xmath1, y s )}, we train a structured prediction model on s, and apply it directly on a target language sentence x t (with ω t , o t ), without seeing any labelled or unlabelled target language data. we will assume a reasonable cross-lingual word representation which maps source and target language words into a same vector space. usually, we use s to estimate probability p(y|x s ) = p(y s |ω s , o s ) which takes word order as an input. models use word order o s as a natural reference to design hyper-parameters (e.g., it sug-gests the linear chain structure of rnn and crf) or extract order sensitive features (e.g., position embeddings in transformer @xcite ). however, the target language may hold a different word order, so this tight connection between word order patterns and model architectures is inflexible for adaptation. here, we propose to estimate p(y, o s |ω s ) instead of p(y|ω s , o s ) (i.e., moving the word order to the output). we can conceptually factorize p(y, o s |ω s ) to p(y|ω s )p(o s |ω s ), where p(y|ω s ) is a structured prediction module, and p(o s |ω s ) is a reordering module whose job is to recover a source language word order from bag-of-words ω (of both source and target language). two modules share internal hidden states and model parameters. comparing with p(y|ω s , o s ), we observe that, • p(y|ω s ) decouples word order and model architectures: we are now free to use orderinsensitive models (e.g., transformer without position embeddings). instead of being in hyperparameters, word order information now appears in the (shared) model parameters of p(y|ω s ) and p(o s |ω s ). when training on the source language, it guides the shared parameters to derive a correct order o s from ω s . when testing on the target language, since words in ω t are in the same space of those in ω s , the learned parameters in p(o s |ω s ) will implicitly encode ω t with a proper source language word order (like what it does for ω s ). as a consequence, p(y|ω t ) will receive a more familiar order information even if o t is different from o s . figure 1 shows an illustration of the two methods. before moving to details of p(y, o s |ω s ), we describe possible representations of word order. ## word order representation to share more parameters with downstream structured prediction tasks, we consider the word order recovery task as another (linear) structured prediction task. for two words w i , w j in the bag of words of a sentence x, we design two kinds of order objectives based on the adjacency matrix, • undirected adjacency matrix m , where m @xmath2. ## joint prediction and reordering we now describe the implementation of p(y, o s |ω s ). basically, it requires the input to be a bag of words without order information. here we choose a transformer encoder without position embeddings foot_0 @xcite instead of order-sensitive networks like rnns. input features denote x 1 , • • • , x n as each word's vector representation. we obtain x i by concatenating a fixed cross-lingual word embedding w i (from muse or mbert), an optional crosslingual part-of-speech tag embedding t i (used only for dependency parsing task), and a trainable occurrence index embedding c i . the c i avoids the isomorphism problem foot_1 of same words by indicating which of them is used first, which is used second, and so on. transformer encoder a transformer encoder is obtained by stacking n layers of identical transformer blocks. we follow the standard notation to introduce how a transformer block outputs a deep representation we start by mapping the input x to three spaces via linear transformations: a self-attention matrix a is then obtained by scaled dot-product attention function. where d is the column dimension of q, a ij measures an interaction score between x i and x j . the self-attention output @xmath3, and the vector o i in o is a deep hidden representation of x i . we can extend the above single-head self-attention layer to multihead by defining multiple sets of {w q , w k , w v }. the output of a multi-head layer is the concate-nation of a set of o k obtained by corresponding q k , k k and v k . we obtain the dimension-reduced output o through a linear transformation: there is a residual connection and normalization (add&norm) layer after the self-attention layer, followed by a feed-forward layer, and finally another residual connection and normalization layer. after these layers we obtain a deep representation the omitted detail of these three layers can be found in @xcite . reordering block a reordering block just adds a reordering signal l order to the ordinary transformer block. as mentioned in section 3, l order can guide the attention matrix a (in equation 1 ), which is also a word-to-word matrix, to display a linear chain structure. after aggregating the shallow representation by using the matrix a as weights, the deep output x of the reordering block has the contextual information with respect to the word order, since a is forced to match the desined word order. we test different setting of l order with different word order repersentation matrices. in our experiments, we will select one of the reordering signals to train the model and compare it with each other. in addition, we use single-head self-attention to reduce computation because preliminary experiments show that multiple heads are not helpful for reordering blocks. @xcite has also shown that replacing multi-head with single-head does not hurt performance. we cross stack reordering blocks with original transformer blocks to build the complete encoder (figure 3 ). the reordering blocks estimate probability p(o s |ω s ) by learning linear word order on the source language. the original blocks estimate probability p(y|ω s ) by learning a structured prediction task without a direct reordering supervision. given the bag of words ω t in target language, thanks to cross-lingual word representation techniques, the reordering block predicts a similar word order p(o s |ω t ) as source language. since the word order of target language is not introduced, the encoder does not have word order drift between target language and source language. downstream classifier for a downstream task, we add a structured prediction classifier over the last encoder block. we investigate three downstream tasks. for graph-based dependency parsing (dep), we follow @xcite to use two bi-affine classifiers. for universal part-ofspeech (upos) and morphological (mor) tagging, we use a multi-layer perceptron (mlp) classifier. we train one of the downstream tasks by the corresponding cross-entropy loss function l @xmath4, l upos , l mor }. we assume the set of reordering block ids is l. the joint objective function l joint is to minimize a weighted combination: ## distillation in equation 2, supervision signals on word reordering blocks are directly obtained from order objective (e.g., -→ m and ← -m ). since we jointly perform reordering and structured prediction. the data for learning word order is constrained by the corpus size of structured prediction task (e.g., treebanks). on the other hand, there are massive unlabelled sentences which can help build a more powerful reordering module. to use those unlabelled data, one challenge is that directly feeding them into the joint learning model could be problematic since the severe imbalance between structured prediction signals and reordering signals would make the model focus on reordering. furthermore, if we solve a different structured prediction task, we need to repeat the learning on those unlabelled data, which could be unnecessary and time-consuming. therefore, it is important to separate the learning on those unsupervised data and sharing them efficiently. here we design a generic p(o s |ω s ) model as a teacher and then distill the knowledge it learned in a large-scale unlabelled corpus into reordering blocks in a structured prediction task, which play the role of a student (figure 4 ). the reordering student model attempts to learn the soft probability distribution from the teacher model, rather than the hard ground truth. furthermore, multiple student models can share one teacher model, which avoids repeated training on a large-scale unlabelled corpus. ## related work there has been a lot of recent research on the cross-lingual transfer of structured prediction tasks, including dependency parsing @xcite , and pos tagging @xcite . early work built delexicalized models to direct transfer, but at the expense of performance @xcite . with the development of cross-lingual word embedding techniques @xcite , the recent work has utilized it to retain lexical information @xcite . data augmentation can enrich the word order that appears on the source language, thereby increasing the intersection with the target language's word order. tiedemann and agic (2016); wang and eisner (2016, 2018a) create a high-quality synthetic treebank to increase source data. but data augmentation requires expert knowledge to build treebank and extra train time. it does not apply to a larger number of target languages. annotation projection relies on cross-language annotation mapping using parallel corpus and automatic alignment @xcite @xcite . our approach does not require the above resources, only the source language's raw data. @xcite ; wang and eisner (2018b); @xcite ; @xcite reorder the source treebanks to make them similar to the target language of interest before training on the source treebanks. this is a sourceto-target reordering that requires the use of parallel corpus or automatic alignment tools. instead, we are target-to-source reordering, and training only at source language. some previous work has seen word order as a trade-off, @xcite modified transformer encoder by using undirected relative position to learn weak order information. @xcite using conv1d to capture local word order and taking the positional embeddings from mbert to initialize a frozen positional embeddings. our reordering module can fully learn the source language word order based on the bag-of-words input. it's useful for tasks that are sensitive to global word order, such as parsing. ## conclusion this work focus on the source-to-target word order adaption in zero-shot transfer. we build structured prediction models containing a novel reordering module with a bag of words input. the reordering module is distilled from a task-generic, unsupervised, and large-scale pre-trained reordering teacher. experiments show that, our model can significantly improve cross-lingual performances on three tasks without obviously hurting source language performance. future work contains two parts: extending to multi-source transfer and extending to more structured prediction tasks such as ner which requires span-level reordering.