Assessing the Generalization Capacity of Pre-trained Language Models through J apanese Adversarial Natural Language Inference (2021)

Abstract: Despite the success of multilingual pre-trained language models, it remains unclear to what extent these models have human-like generalization capacity across languages. The aim of this study is to investigate the out-of-distribution generalization of pre-trained language models through Natural Language Inference (NLI) in Japanese, the typological properties of which are different from those of English. We introduce a synthetically generated Japanese NLI dataset, called the Japanese Adversarial NLI (JaNLI) dataset, which is inspired by the English HANS dataset and is designed to require understanding of Japanese linguistic phenomena and illuminate the vulnerabilities of models. Through a series of experiments to evaluate the generalization performance of both Japanese and multilingual BERT models, we demonstrate that there is much room to improve current models trained on Japanese NLI tasks. Furthermore, a comparison of human performance and model performance on the different types of garden-path sentences in the JaNLI dataset shows that structural phenomena that ease interpretation of garden-path sentences for human readers do not help models in the same way, highlighting a difference between human readers and the models.

## introduction generalization is one of the essential components that account for the understanding of language. in recent years, pre-trained models such as bert @xcite have provided high performance on both english benchmarks @xcite and multilingual benchmarks @xcite , suggesting that they might have some cross-lingual generalization capacity. yet, while these models have achieved remarkable performance on an indistribution test set (i.e., training and test splits are given as the same distribution), several previous studies have pointed out that the models fail on out-of-distribution test sets (i.e., examples drawn from a distribution different from that of the training set) @xcite and that the models varied widely in terms of the generalization performance @xcite . it remains an open question to what extent pre-trained models can realize human-like generalization ability. a standard task for assessing whether pre-trained language models possess human-like language understanding is natural language inference (nli), which is the task of judging whether a premise sentence entails a hypothesis sentence. recently, a number of studies have sought to probe the generalization performance of models and detected their fallible heuristics with various nli datasets @xcite @xcite @xcite . however, these studies tend to focus on english nli datasets, and independent analysis in multiple languages would be desirable. in response to this challenge, the study of the outof-distribution generalization ability of nli models from cross-lingual perspectives has begun to be explored @xcite but is not yet fully developed. the aim of this paper is to investigate to what extent pre-trained language models have generalization capacity in japanese nli. for this purpose, we present a japanese linguistically challenging nli dataset, called the japanese adversarial nli (janli) dataset. 1 this dataset is inspired by the english hans dataset @xcite and is designed to cover a variety of linguistic phenomena specific to japanese, a language typologically different from english @xcite . generating inference examples in a controlled way allows us to analyze whether language models are sensitive to factors such as word order and syntactic structure in japanese. we present a series of experiments with the janli dataset to evaluate the generalization performance of japanese and multilingual bert models. in addition, we compare human performance with model performance. our experiments shed light on several shortcomings of the models and highlight the following challenges for cross-lingual generalization in nli: ## related work previous studies have probed pre-trained language models on various nli tasks and discovered that generalization capacity is limited for understanding diverse linguistic phenomena @xcite @xcite @xcite @xcite and annotation artifacts @xcite in standard english nli datasets such as the snli @xcite and multinli @xcite datasets. the work most closely related to ours is hans @xcite , which is an nli dataset designed to analyze whether models use structural heuristics to make predictions. recently, hans has been used for out-of-distribution evaluation data @xcite @xcite and has been used for data augmentation to improve the generalization performance of models @xcite . although the generalization capacity of nli models has been studied mainly in english, non-english nli datasets @xcite @xcite and multilingual nli datasets @xcite have recently been developed to analyze the performance of pre-trained language models across languages. several japanese nli datasets have been created. the japanese snli dataset @xcite was generated by automatic translation of the english snli dataset. the japanese realistic textual entailment corpus @xcite was created by using realistic texts (hotel reviews) and annotating labels via crowdsourcing. jsem @xcite is a japanese version of the fracas test suite @xcite , which contains manually designed problems involving semantic phenomena that have been well studied in formal semantics. our dataset is designed to assess whether models capture linguistic structures in japanese or simply rely on fallible heuristics. recent work @xcite @xcite has shown that shuffled word order has little effect during training or inference with pre-trained language models, which in turn indicates that the models are insensitive to word order in nli tasks. in english, however, the shuffled data are usually unacceptable and tend to obscure their gold labels. @xcite have re-analyzed the hypothesis that language models with lower perplexity are more human-like language models in japanese rather than in english, and their experiments have demonstrated the lack of universality of this hypothesis and the importance of cross-lingual evaluation of models. japanese word order is fairly free, which enables us to produce grammatically correct sentences even when the word order is shuffled. analyzing the behavior of models on controlled japanese inference examples should provide further insights into the sensitivity of the models to word order. ## dataset generation to analyze the generalization capacity of nli models, we introduce a synthetically generated japanese nli dataset where each pair (p, h) of a premise and hypothesis is tagged with a label for structural pattern and linguistic phenomenon. constituent h is a constituent of p. phenomena: modal p: ひょっとしたら 子供 が 眠っている perhaps child ga sleeping (perhaps the child is sleeping) h: 子供 が 眠っている child ga sleeping (the child is sleeping) ## conclusion we introduced the janli dataset, which was designed to assess the generalization capacity of pretrained language models on nli in japanese. experiments showed that both japanese and multilingual bert models trained with basic japanese nli datasets performed very poorly on the janli dataset. in addition, both japanese and multilingual models, in particular the latter, struggled with learning some japanese linguistic phenomena even when augmented with a portion of the janli dataset. this suggests that there is still much room for improving the generalization capacity of pre-trained language models. lastly, the comparison between human performance and model performance illustrated that whereas the models failed to correctly predict labels for non-entailment examples, human judgement was near perfect. furthermore, factors that ease the interpretation of gardenpath sentences for humans do not help model predictions. overall, our dataset illuminates the vulnerabilities of the currently standard pre-trained language models and indicate a new challenge for cross-lingual generalization of nli. takumi yoshikoshi, daisuke kawahara, and sadao kurohashi. 2020. multilingualization of natural language inference datasets using machine translation (in japanese). in proceedings of the 244th meeting of natural language processing.