Compositional Evaluation on J apanese Textual Entailment and Similarity (2022)

Abstract: Natural Language Inference (NLI) and Semantic Textual Similarity (STS) are widely used benchmark tasks for compositional evaluation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclusively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically different from English and can shed light on the currently controversial behavior of language models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. We also present a stress-test dataset for compositional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on different pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking.

## introduction natural language inference (nli) @xcite and semantic textual similarity (sts) @xcite tasks are well positioned to serve as a basic benchmark for natural language understanding. with the recent progress of deep neural networks, including pre-trained language models such as bert @xcite , the development of benchmark datasets has centered on large crowdsourced english datasets, such as snli @xcite and multinli @xcite . since there has been an increasing need for benchmark datasets in linguistic universals (linzen, 2020), general language understanding frameworks including nli and sts for languages other than english have been provided @xcite @xcite @xcite @xcite . another recent line of work has investigated whether models are sensitive to shuffled word order, but the conclusions are controversial @xcite @xcite @xcite . one characteristic of human-like language understanding is that humans can understand sentences according to their word meanings and syntactic structures, then recognize their semantic relationships @xcite @xcite . since previous work has demonstrated the usefulness of analyzing the generalization ability of models in challenging nli in english @xcite @xcite @xcite , we should continue this line of research in other languages. against this background, we provide a japanese nli/sts dataset to analyze language models in compositional inference across languages. our motivations for focusing on japanese are two-fold. first, japanese is a high-resource language that has typologically different characteristics from english @xcite ), yet it has not been included in previous cross-lingual @xcite @xcite or multilingual @xcite nli datasets. this raises the question of whether models perform inference differently in japanese and other languages. second, japanese has case markers and free word-order @xcite , phenomena that pose interesting challenges for multilingual nli. while shuffling data usually changes its meaning, the meaning of a japanese sentence can be preserved even when the order of noun phrase (np) arguments is swapped. by analyzing model behavior with scrambling phenomena that preserve case relations in a sentence and particle-swapping phenomena that change case relations, we can analyze whether the model can distinguish transformations that change sentence meanings and perform compositional inferences. this paper has three contributions. first, we provide jsick as a compositional japanese nli/sts dataset by manually translating the english sick dataset @xcite . compared with recent crowdsourced nli datasets, sick facilitates identification of which compositional linguistic phenomena are key to a given inference. such a controlled structure is suited to transforming sentences for further analyses of model behavior. in addition, sick has been translated into non-english languages @xcite , allowing cross-language comparisons on a sizeable parallel nli corpus. second, we create a stress-test dataset for jsick to investigate whether language models capture word order and case particles in japanese. we created the stress-test dataset by transforming syntactic structures of jsick sentence pairs, where we analyze whether models consider word order and case particles when predicting entailment labels and similarity scores. third, for the baseline evaluation of pre-trained language models, we compare performance between different pre-trained language models on jsick. we also compare the performance of multilingual pre-trained language models on sick datasets of different languages, including jsick. we also provide an in-depth analysis of sensitivity to word order and case particles based on the jsick stress-test dataset. the analysis results suggest that both japanese and multilingual models are surprisingly inattentive to word order and case marking. our dataset is publicly available at https://github.com /verypluming/jsick . with the development of multilingual pretrained language models, general language understanding frameworks for languages other than english have been created @xcite @xcite @xcite , and nli datasets have been multilingualized. @xcite provided a cross-lingual nli (xnli) corpus by translating multinli into 15 languages, including languages with few language resources such as swahili and urdu and languages with flexible word order such as russian and german. @xcite translated multinli into korean to create kornli. however, japanese is not included in these datasets. in addition, since sentences in multinli are usually longer than those in sick and contain multiword expressions beyond the scope of compositional inference, it is unrealistic to carefully transform syntactic structures of sentence pairs in xnli to create a stress-test dataset like ours. as examples of other non-english datasets, ocnli @xcite is a chinese nli dataset built from original multi-genre resources. farstail @xcite ) is a persian nli dataset containing sentences from university exams. there have also been attempts to translate the sick dataset into portuguese @xcite and dutch @xcite , so our japanese sick dataset will contribute to a multilingual sick dataset that will allow controlled, cross-lingual analyses of the compositional abilities of language models. regarding japanese nli datasets, a japanese snli dataset @xcite was constructed by using machine translation to translate the english snli dataset into japanese and automatically filtering out unnatural sentences, but methods that employ machine translation are still problematic in that they can produce unnatural sentences. the japanese realistic textual entailment corpus @xcite ) is a crowdsourced dataset containing japanese hotel reviews. however, linguistic phenomena in these japanese datasets demonstrate limited diversity because the sentences they contain are restricted to simple structures. @xcite provided jsem, a manually curated test set including a japanese version of fracas to diagnose inference systems from a formal semantics perspective. we produced an nli dataset by asking experts to translate the sick dataset into japanese, thus maintaining both sentence naturalness and compositions of linguistic phenomena. while recent works @xcite @xcite have shown that pre-trained language models are insensitive to word order on permuted english datasets in the standard natural language understanding benchmark glue @xcite including nli, other works have analyzed sentence perplexity with varying word orders and shown controversial results regarding inductive biases for word order in different languages @xcite . for six languages, including japanese, @xcite evaluated whether multilingual bert captures word order in the translated paws dataset @xcite , involving adversarial paraphrase identification pairs whose sentences share words but differ in word order. experiments showed that bert performance for japanese is consistently worse than that for indo-european languages. our study deepens insight into the causes of performance differences by stress-test evaluation of nli and sts tasks through careful manipulation of case markers, which are more challenging tasks than are two-class paraphrase identification tasks. @xcite reexamined the general hypothesis that language models with lower perplexity are more human-like in japanese than in english, and the results have shown the necessity of evaluating models across languages. analyzing the model behavior, rather than perplexity, in transformed japanese inference should provide new insights into the model's sensitivity to word order. ## conclusion we introduced jsick, a japanese standard nli/sts dataset, by manually translating english sick into japanese and re-annotating its gold labels. in baseline experiments, we compared the performance of various pre-trained japanese language models on jsick. while the japanese roberta-large model achieved state-of-the-art performance, the performance of multilingual pre-trained language models achieved comparable results. experiments with multilingual models on sick datasets in different languages, including jsick, showed that the performance of multilingual models was relatively low on inference problems involving anaphora, disjunction, and additive particles. furthermore, to investigate the extent to which japanese and multilingual pre-trained language models are sensitive to word order and case particles, we provided a jsick stress-test dataset involving word scrambling and particle-swapping from jsick. the results from that dataset suggest that both japanese and multilingual models do not consider word order and case particles when making predictions for japanese nli/sts tasks. these are novel findings that are not obtainable from other datasets, including sick in english and other languages. overall, the results suggest large room for improvement of both japanese and multilingual pre-trained language models regarding their sensitivity to flexible word order and the representations of case particles. further improvements might be obtained by more adequate representations of japanese vocabularies in multilingual pre-trained language models @xcite . we believe our dataset will be useful in future research for realizing more advanced models capable of appropriately performing multilingual compositional inference.