Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in J apanese (2020)

Abstract: We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LM-based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LM-based method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by large-scale experiments.

## introduction speakers sometimes have a range of options for word order in conveying a similar meaning. a typical case in english is dative alternation: ## linguistic background this section provides a brief overview of the linguistic background of canonical word order, some basics of japanese grammar, and common methods of linguistic analysis. ## lm-based method 3.1 overview of the lm-based method in the nlp field, lms are widely used to estimate the acceptability of text @xcite . an overview of the lm-based method is shown in figure 1 . after preparing several word orders considering the targeted linguistic hypothesis, we compare their generation probabilities in lms. we assume that the word order with the highest generation probability follows their canonical word order. ## experiment1: comparing human and lms word order preference to examine the validity of using lms for canonical word order analysis, we examined the parallels between the lms and humans on the task determining the canonicality of the word order (figure 2 ). first, we created data for this task (section 4.1). we then compared the word order preference of lms and that of humans (section 4.2). ## experiment2: consistency with previous studies this section examines whether lms show word order preference consistent with previous linguistic studies. the results are entirely consistent, which support the validity of the lm-based methods in japanese. each subsection focuses on a specific component of japanese sentences. ## analysis: word order and topicalization in the previous section, we tentatively concluded that lms can be used for analyzing the intrasentential properties on the canonical word order. based on this finding, in this section, we demonstrate the analysis of additional claims on the properties of the canonical word order with the lmbased method, which has been less explored by large-scale experiments. this section shows the analysis of the relationship between topicalization and the canonical word order. additional analyses on the effect of various adverbial particles for the word order are shown in appendix f. ## conclusion and future work we have proposed to use lms as a tool for analyzing word order in japanese. our experimental results support the validity of using japanese lms for canonical word order analysis, which has the potential to broaden the possibilities of linguistic research. from an engineering view, this study supports the use of lms for scoring japanese word order automatically. from the viewpoint of the linguistic field, we provide additional empirical evidence to various word order hypotheses as well as demonstrate the validity of the lm-based method. we plan to further explore the capability of lms on other linguistic phenomena related to word order, such as "given new ordering" @xcite . since lms are language-agnostic, analyzing word order in another language with the lm-based method would also be an interesting direction to investigate. furthermore, we would like to extend a comparison between machine and human language processing beyond the perspective of word order.