Sequence Mixup for Zero-Shot Cross-Lingual Part-Of-Speech Tagging (2021)

Abstract: There have been efforts in cross-lingual transfer learning for various tasks. We present an approach utilizing an interpolative data augmentation method, Mixup, to improve the generalizability of models for part-of-speech tagging trained on a source language, improving its performance on unseen target languages. Through experiments on ten languages with diverse structures and language roots, we put forward its applicability for downstream zero-shot cross-lingual tasks.

## introduction recently, neural network models have obtained state-of-the-art results in part-of-speech (pos) tagging tasks across multiple languages. since numerous languages lack suitable corpora annotated with pos labels, there have been efforts to design models for cross-lingual transfer learning. crosslingual learning enables us to utilize the annotated corpora of a source language to train models that are effective over a different target language. interpolative data augmentation methods have been proposed to mitigate overfitting in models in the absence of enough training data. sequence-based mixup @xcite is an interpolative data augmentation method for named entity recognition. however, these methods have not been explored for cross-lingual transferability. interpolative data augmentation methods are aimed at increasing the diversity of the training distribution and, as a result, improving the generalizability of underlying models. we leverage this capability of sequence mixup (seq. mixup) to capture rich linguistic information for cross-lingual transferability of pos tagging tasks for ten languages with different structures and language roots. to this end, we first measure the dataset level cosine similarity across languages, defined as the average of sentence-level embedding of dataset samples. this gives an overview of the syntactical and semantical relationship among the different languages. we then finetune multilingual models over a source language using sequence mixup and evaluate it across a target language for varying language similarities, probing sequence-based interpolative data augmentation. we also evaluate sequence mixup on combinations of similar and dissimilar languages and verify its transferability on various target languages. ## methodology mixup @xcite is a data augmentation technique that generates virtual training samples from convex combinations of individual inputs and labels. for a pair of data points (x, y) and (x , y ), mixup creates a new sample ( x, y) by interpolating the data points using a ratio λ, sampled from a beta distribution, where @xmath0. we perform mixup over the latent space representations for interpolating sequences. pair of sentences (x, y) and (x , y ) are randomly sampled and interpolated in the hidden space using a l-layer encoder f (:, θ). the hidden layer representations for x and x upto the k th layer are given as, at the k th layer, the hidden representations of each token in x are linearly interpolated with each token in x . after this, h k is fed to the upper layers, we evaluate the performance of zero-shot learning on sequence mixup, where the model is trained on one source language or a set of source languages and evaluated on a target language. to choose the source and target, for each language we average sentence level embeddings of dataset instances and use the average embeddings as language representation. using these representations, we find the cosine similarity among the languages as shown in ## experiments we evaluate our approach on pos tagging with datasets from the universal dependencies (ud) dataset 1 for ten different languages -arabic (ar), dutch (nl), french (fr), german (de), hindi (hi), indonesian (id), italian (it), marathi (mr), vietnamese (vi) and urdu (ur). for each experiment, we use 800 sentences for each source language for training, 100 sentences for validation and test each from the target language for evaluation. bert-basemultilingual-cased (mbert) has been used as an encoder in sequence mixup and for obtaining the embeddings to evaluate cosine similarity. ## conclusion we analyze interpolative regularization based data augmentation over tokens for zero-shot crosslingual transfer of part-of-speech tagging across ten languages. through extensive experiments over languages with varying syntactic and semantic structures on single and pair of languages, we pave the way for using interpolative data augmentation to improve the generalizability of neural networks for zero-shot transfer learning on downstream tasks.