Syntax-augmented Multilingual BERT for Cross-lingual Transfer (2021)

Abstract: In recent years, we have seen a colossal effort in pre-training multilingual text encoders using large-scale corpora in many languages to facilitate cross-lingual transfer learning. However, due to typological differences across languages, the cross-lingual transfer is challenging. Nevertheless, language syntax, e.g., syntactic dependencies, can bridge the typological gap. Previous works have shown that pre-trained multilingual encoders, such as mBERT (CITATION), capture language syntax, helping cross-lingual transfer. This work shows that explicitly providing language syntax and training mBERT using an auxiliary objective to encode the universal dependency tree structure helps cross-lingual transfer. We perform rigorous experiments on four NLP tasks, including text classification, question answering, named entity recognition, and task-oriented semantic parsing. The experiment results show that syntax-augmented mBERT improves cross-lingual transfer on popular benchmarks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across all languages. In the generalized transfer setting, the performance boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA.

## introduction cross-lingual transfer reduces the requirement of labeled data to perform natural language processing (nlp) in a target language, and thus has the ability to avail nlp applications in low-resource languages. however, transferring across languages is challenging because of linguistic differences at levels of morphology, syntax, and semantics. for example, word order difference is one of the crucial factors that impact cross-lingual transfer @xcite . the two sentences in english and hindi, as shown in figure 1 have the same * work done during internship at facebook ai. figure 1 : two parallel sentences in english and hindi from xnli @xcite dataset. the words highlighted with the same color have the same meaning. although the sentences have a different word order, their syntactic dependency structure is similar. meaning but a different word order (while english has an svo (subject-verb-object) order, hindi follows sov). however, the sentences have a similar dependency structure, and the constituent words have similar part-of-speech tags. presumably, language syntax can help to bridge the typological differences across languages. in recent years, we have seen a colossal effort to pre-train transformer encoder @xcite on large-scale unlabeled text data in one or many languages. multilingual encoders, such as mbert @xcite or xlm-r @xcite ) map text sequences into a shared multilingual space by jointly pre-training in many languages. this allows us to transfer the multilingual encoders across languages and have found effective for many nlp applications, including text classification @xcite english how many members of the senate are elected? spanish cuántos miembros del senado son elegidos? c english the chamber of deputies has 630 elected members, while the senate has 315 elected members. . . . spanish la cámara de los diputados está formada por 630 miembros, mientras que hay 315 senadores más los senadores vitalicios. . . . a mbert [q:english-c:english] 315 (); [q:spanish-c:spanish] 630 () [q:spanish-c:english] 315 (); [q:english-c:spanish] 630 () mbert + syn. a few recent works @xcite @xcite suggest that bert learns compositional features; mimicking a tree-like structure that agrees with the universal dependencies taxonomy. however, fine-tuning for the downstream task in a source language may not require mbert to retain structural features or learn to encode syntax. we argue that encouraging mbert to learn the correlation between syntax structure and target labels can benefit cross-lingual transfer. to support our argument, we show an example of question answering (qa) in figure 2 . in the example, mbert predicts incorrect answers given the spanish language context that can be corrected by exploiting syntactic clues. utilizing syntax structure can also benefit generalized cross-lingual transfer @xcite where the input text sequences belong to different languages. for example, answering an english question based on a spanish passage or predicting text similarity given the two sentences as shown in figure 1 . in such a setting, syntactic clues may help to align sentences. in this work, we propose to augment mbert with universal language syntax while fine-tuning on downstream tasks. we use a graph attention network (gat) @xcite to learn structured representations of the input sequences that are incorporated into the self-attention mechanism. we adopt an auxiliary objective to train gat such that it embeds the dependency structure of the input sequence accurately. we perform an evaluation on zero-shot cross-lingual transfer for text classification, question answering, named entity recognition, and task-oriented semantic parsing. experiment results show that augmenting mbert with syntax improves cross-lingual transfer, such as in paws-x and mlqa, by 1.4 and 1.6 points on average across all the target languages. syntaxaugmented mbert achieves remarkable gain in the generalized cross-lingual transfer; in paws-x and mlqa, performance is boosted by 3.9 and 3.1 points on average across all language pairs. furthermore, we discuss challenges and limitations in modeling universal language syntax. we release the code to help future works. foot_0 2 syntax-augmented multilingual bert multilingual bert (mbert) @xcite enables cross-lingual learning as it embeds text sequences into a shared multilingual space. mbert is fine-tuned on downstream tasks, e.g., text classification using monolingual data and then directly employed to perform on the target languages. this refers to zero-shot cross-lingual transfer. our main idea is to augment mbert with language syntax for zero-shot cross-lingual transfer. we employ graph attention network (gat) @xcite to learn syntax representations and fuse them into the self-attention mechanism of mbert. in this section, we first briefly review the transformer encoder that bases mbert ( § 2.1), and then describe the graph attention network (gat) that learns syntax representations from dependency structure of text sequences ( § 2.2). finally, we describe how language syntax is explicitly incorporated into the transformer encoder ( § 2.3). ## experiment setup to study syntax-augmented mbert's performance in a broader context, we perform an evaluation on four nlp applications: text classification, named entity recognition, question answering, and taskoriented semantic parsing. our evaluation focuses on assessing the usefulness of utilizing universal syntax in the zero-shot cross-lingual transfer. ## experiment results we aim to address the following questions. 1. does augmenting mbert with syntax improve (generalized) cross-lingual transfer? 2. does incorporating syntax benefit specific languages or language families? 3. which nlp tasks or types of tasks get more benefits from utilizing syntax? ## conclusion in this work, we propose incorporating universal language syntax into multilingual bert (mbert) by infusing structured representations into its multihead attention mechanism. we employ a modified graph attention network to encode the syntax structure of the input sequences. the results endorse the effectiveness of our proposed approach in the cross-lingual transfer. we discuss limitations and challenges to drive future works.