Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion (2022)

Abstract: Incorporating stronger syntactic biases into neural language models (LMs) is a long-standing goal, but research in this area often focuses on modeling English text, where constituent treebanks are readily available. Extending constituent tree-based LMs to the multilingual setting, where dependency treebanks are more common, is possible via dependency-to-constituency conversion methods. However, this raises the question of which tree formats are best for learning the model, and for which languages. We investigate this question by training recurrent neural network grammars (RNNGs) using various conversion methods, and evaluating them empirically in a multilingual setting. We examine the effect on LM performance across nine conversion methods and five languages through seven types of syntactic tests. On average, the performance of our best model represents a 19 % increase in accuracy over the worst choice across all languages. Our best model shows the advantage over sequential/overparameterized LMs, suggesting the positive effect of syntax injection in a multilingual setting. Our experiments highlight the importance of choosing the right tree formalism, and provide insights into making an informed decision.

## introduction the importance of language modeling in recent years has grown considerably, as methods based on large pre-trained neural language models (lms) have become the state-of-the-art for many problems @xcite . however, these neural lms are based on general architectures and therefore do not explicitly model linguistic constraints, and have been shown to capture only a subset of the syntactic representations typically found in constituency treebanks @xcite ). an alternative line of lm research aims to explicitly model the parse tree in order to make the lm syntax-aware. a representative example of this paradigm, reccurent neural network grammar @xcite , is reported to perform better than sequential lms on tasks that require complex syntactic analysis @xcite @xcite . the aim of this paper is to extend lms that inject syntax to the multilingual setting. this attempt is important mainly in two ways. firstly, english has been dominant in researches on syntax-aware lm. while multilingual lms have received increasing attention in recent years, most of their approaches do not explicitly model syntax, such as multilingual bert (mbert, @xcite or xlm-r @xcite . although these models have shown high performance on some cross-lingual tasks @xcite , they perform poorly on a syntactic task @xcite . secondly, syntax-aware lms have interesting features other than their high syntactic ability. one example is the validity of rnng as a cognitive model under an english-based setting, as demonstrated in @xcite . since human cognitive functions are universal, while natural languages are diverse, it would be ideal to conduct this experiment based on multiple languages. the main obstacle for multilingual syntax-aware modeling is that it is unclear how to inject syntactic information while training. a straightforward approach is to make use of a multilingual treebank, such as universal dependencies (ud, @xcite , where trees are represented in a dependency tree (dtree) formalism. @xcite evaluated parsing and language modeling performance on three typologically different languages, using a generative dependency model. unfortunately, they revealed that dependency-based models are less suited to language modeling than comparable constituencybased models, highlighting the apparent difficulty of extending syntax-aware lms to other languages using existing resources. this paper revisits the issue of the difficulty of constructing multilingual syntax-aware lms, by exploring the performance of multilingual language modeling using constituency-based models. since our domain is a multilingual setting, our focus turns to how dependency-to-constituency conversion techniques result in different trees, and how these trees affect the model's performance. we obtain constituency treebanks from ud-formatted dependency treebanks of five languages using nine tree conversion methods. these treebanks are in turn used to train an rnng, which we evaluate on perplexity and clams @xcite . our contributions are: (1) we propose a methodology for training multilingual syntax-aware lms through the dependency tree conversion. (2) we found an optimal structure that brings out the potential of rnng across five languages. (3) we demonstrated the advantage of our multilingual rnng over sequential/overparameterized lms. ## method: dependency tree conversion as a source of multilingual syntactic information, we use universal dependencies (ud), a collection of cross-linguistic dependency treebanks with a consistent annotation scheme. since rnng requires a ctree-formatted dataset for training, we perform dtree-to-ctree conversions, which are completely algorithmic to make it work regardless of language. our method consists of two procedures: structural conversion and nonterminal labeling; obtaining a ctree skeleton with unlabeled nonterminal nodes, then assigning labels by leveraging syntactic information contained in the dependency annotations. while our structural conversion is identical to the baseline approach of @xcite , we include a novel labeling method that relies on dependency relations, not pos tags. ## what is the robust conversion which works well in every language? in section 3, we proposed language-independent multiple conversions from dtree to ctree. the intriguing question is; is there a robust conversion that brings out the potential of rnng in every language? to answer this question, we conducted a thorough experiment to compare the performances of rnngs trained in each setting. ## advantage of syntax injection to lms in a multilingual setting in this section, we demonstrate the benefits of injecting syntactic biases into the model in a multilingual setting. we obtained the clams score of rnng trained on the flat-dep treebank (flat-dep rnng for short) and compared it against baselines. ## discussion we discussed the ctree structure that works robustly regardless of the language and the superiority of injecting syntactic bias to the model. our claim is that we can construct languageindependent syntax-aware lms by seeking the best structure for learning rnngs, which is backed up by our experiments based on five languages. to make this claim firm, more investigations are needed from two aspects: fine-grained syntactic evaluation and experiment on typologically diverse languages. fine-grained syntactic evaluation the linguistic phenomenon covered in clams is only an agreement. however, previous works have invented evaluation sets that examine more diverse syntactic phenomena for english @xcite . we need such a fine-grained evaluation even in a multilingual setting, as superiority in agreement does not imply superiority in every syntactic knowledge; @xcite suggested that rnng performs poorer than lstm lm in capturing sentential complement or simple negative polarity items. it is challenging to design a multiliugnal syntactic test set because even an agreement based on grammatical categories is not a universal phenomenon. it is required to seek reasonable metrics that cover broad syntactic phenomena and are applicable to many languages. ## conclusion in this paper, we propose a methodology to learn multilingual rnng through dependency tree conversion. we performed multiple conversions to seek the robust structure which works well multilingually, discussing the effect of multiple structures. we demonstrated the superiority of our model over baselines in capturing syntax in a multilingual setting. since our research is the first step for multilingual syntax-aware lms, it is necessary to conduct experiments on more diverse languages to seek a better structure. we believe that this research would contribute to the field of theoretical/cognitive linguistics as well because an ultimate goal of linguistics is finding the universal rule of natural language. finding a reasonable structure in engineering would yield useful knowledge for that purpose.