Struct- XLM : A Structure Discovery Multilingual Language Model for Enhancing Cross-lingual Transfer through Reinforcement Learning (2023)

Abstract: Cross-lingual transfer learning heavily relies on well-aligned cross-lingual representations. The syntactic structure is recognized as beneficial for cross-lingual transfer, but limited researches utilize it for aligning representation in multilingual pre-trained language models (PLMs). Additionally, existing methods require syntactic labels that are difficult to obtain and of poor quality for low-resource languages. To address this gap, we propose Struct-XLM, a novel multilingual language model that leverages reinforcement learning (RL) to autonomously discover universal syntactic structures for improving the cross-lingual representation alignment of PLM. Struct-XLM integrates a policy network (PNet) and a translation ranking task. The PNet is designed to discover structural information and integrate it into the last layer of the PLM through the structural multi-head attention module to obtain structural representation. The translation ranking task obtains a delayed reward based on the structural representation to optimize the PNet while improving the alignment of cross-lingual representation. Experiments show the effectiveness of the proposed approach for enhancing cross-lingual transfer of multilingual PLM on the XTREME benchmark.

## introduction cross-lingual representation is a crucial component in the development of multilingual natural language processing (nlp) systems @xcite . various multilingual pre-trained language models (plms) like xlm-r @xcite , xy-lent @xcite , and veco2.0 @xcite have demonstrated their effectiveness in cross-lingual transfer learning. the alignment of cross-lingual representation is vital for effective cross-lingual transfer learn- † corresponding authors. 1 the code is available at https://github.com/ wulinjuan/struct-xlm ing. previous researches extend the pre-training tasks by introducing novel pre-training objectives at different levels of granularity, such as token-level @xcite , word-level @xcite , and sentencelevel @xcite @xcite . these objectives provide explicit guidance for the model to learn aligned information across languages. some models @xcite proposed cross-lingual attention mechanisms to aligned cross-lingual representation. however, there is still limited research on aligning crosslingual representations with syntactic structurelevel information. the performance of cross-lingual transfer is also influenced by structural differences among languages @xcite . and the incorporation of structural information, particularly syntax, has proven beneficial. syntaxaugmented mbert @xcite proposed a graph attention network (gan) to learn universal dependency tree labels and integrate them into the self-attention mechanism of mbert for enhancing cross-lingual transfer. the utilization of syntactic structures also demonstrates improvements in various cross-lingual tasks, such as crosslingual structured sentiment analysis @xcite , cross-lingual semantic role labeling @xcite , and cross-lingual word sense disambiguation @xcite . however, existing methods heavily rely on dependency treebanks or syntactic tagging tools to obtain syntactic labels, which are often limited in quantity or quality, particularly for low-resource languages. in this paper, we propose struct-xlm, a multilingual language model with structure discovery based on multilingual plm, that aims to improve the alignment of cross-lingual representations of plm by automatically learning universal structural information, such as constituent syntactic structure. as shown in figure 1 , we employ reinforcement learning (rl) to discover these syntactic structures without explicit structure annotations. following @xcite , we formulate the structure discovery as a sequential decision problem optimized through policy gradient and implemented by a masked self-attention mechanism. additionally, we introduce a structural multi-head attention mechanism, as shown in figure 2 , to effectively integrate the acquired structural information into the last layer of the multilingual pre-trained lm. our model follows a siamese framework and is trained using a policy network (pnet) and translation ranking task, in which the latter provides delayed rewards for rl while enhancing crosslingual alignment. during the fine-tuning phase for downstream tasks, the parameters of the pnet are kept frozen. to summarize, our main contributions are as follows. ## conclusion in conclusion, we introduced struct-xlm, a novel approach to enhance cross-lingual transfer learning in multilingual language models. by leveraging reinforcement learning without explicit structure annotations, the struct-xlm discovers structures and improves the alignment of cross-lingual representations. specifically, we employ policy gradient rl and a translation ranking task to discover constituent syntactic structures and enhance crosslingual transfer. experimental results on seven xtreme benchmark tasks demonstrated the effectiveness of struct-xlm, outperforming baseline plm models with an average improvement of 4.1 points and being competitive with the best strong baseline with small training data. our analysis also revealed that struct-xlm improves performance on the sentence retrieval task bucc, particularly in the middle layers of the pre-trained model, indicating the acquisition of valuable structural information for better-aligned representations.