Can Large Language Models Robustly Perform Natural Language Inference for J apanese Comparatives? (2025)

Abstract: Large Language Models (LLMs) perform remarkably well in Natural Language Inference (NLI).However, NLI involving numerical and logical expressions remains challenging.Comparatives are a key linguistic phenomenon related to such inference, but the robustness of LLMs in handling them, especially in languages that are not dominant in the models’ training data, such as Japanese, has not been sufficiently explored.To address this gap, we construct a Japanese NLI dataset that focuses on comparatives and evaluate various LLMs in zero-shot and few-shot settings.Our results show that the performance of the models is sensitive to the prompt formats in the zero-shot setting and influenced by the gold labels in the few-shot examples.The LLMs also struggle to handle linguistic phenomena unique to Japanese.Furthermore, we observe that prompts containing logical semantic representations help the models predict the correct labels for inference problems that they struggle to solve even with few-shot examples.

## introduction in recent years, large language models (llms) have demonstrated high performance across a wide range of tasks, including natural language inference (nli; @xcite . however, inference with numerical and logical expressions remains challenging for llms @xcite @xcite . in particular, nli involving comparatives is important, as it requires a proper understanding of such expressions. indeed, there are english benchmarks focusing on comparatives for pre-trained models and inference systems @xcite ). however, it has not been thoroughly investigated how robust llms are in handling various types of inference involving comparatives, regardless of the prompt formats or the few-shot example selection. moreover, there is growing attention to analyzing the robustness of inference in languages that are not dominant in the pre-training data. given these motivations, we construct an nli dataset focusing on japanese comparatives by creating templates from an existing japanese nli dataset and filling in them with words. 1 using this dataset, we evaluate five llms, including both open and commercial models. we analyze how robustly llms can perform inference on comparatives regardless of the way prompts are given in zero-shot and few-shot settings. we also compare llms with ccg-jcomp foot_1 (mikami et al. 2025), a logical inference system for japanese comparatives. the experimental results suggest that the prompt formats impact the model behavior in the zero-shot settings, and that the few-shot performance is influenced by the gold labels in the few-shot examples. in addition, prompts with semantic representations from ccg-jcomp can improve model accuracy on problems that remain difficult even with standard few-shot settings. ## related work in this section, we describe existing datasets that contain inference problems involving comparatives. jsem @xcite ) is a japanese nli dataset, constructed from the english nli dataset fracas @xcite with some additional problems that cover inference unique to japanese. the problems are divided into sections based on semantic phenomena, including comparatives, which allows us to evaluate the strengths and weaknesses of models with respect to individual phenomena. however, since jsem is limited in vocabulary and small in scale, we create templates from the dataset and generate new problems by filling in the templates with various words. cad @xcite ) is a dataset on english adjectives, comparatives, adverbs, and quantifiers. the authors chose inference examples from linguistic papers and constructed new problems by applying transformations such as adding negation and replacing words. adjective scale probe @xcite ) is a dataset designed to investigate how well language models understand degree semantics. it is semi-automatically generated based on templates. while these studies evaluate the extent to which pre-trained language models perform inference involving comparatives in fine-tuned settings, they do not specifically focus on the robustness of the inference in in-context learning settings. to address this gap, we provide a scalable nli dataset involving japanese comparatives based on templates created from existing hand-crafted nli problems. ## dataset creation to analyze the extent to which llms robustly perform inference involving japanese comparatives, we create an nli dataset based on the comparatives section of jsem. our dataset construction process is composed of (i) template creation based on jsem and (ii) problem creation using the templates. ## evaluation of zero-shot nli first, we analyze how consistent the performance of the llms is regardless of the prompts in the zero-shot prompt setting, compared with a logical inference system. ## conclusion in this study, we constructed an nli dataset focusing on japanese comparatives, and analyzed how robustly llms can perform inference involving comparatives in zero-shot and few-shot settings. the zero-shot experiment revealed that the models' performance varies depending on the prompts, and each model exhibited a distinctive pattern of errors. in the few-shot experiments, we observed that some models, such as swallow70b-inst, showed a decrease in accuracy when given adversarially designed examples. this observation suggests that some models may be overly sensitive to the specific labels included in the few-shot examples. for problems that the models struggled to solve in the few-shot settings, we found that the accuracy can be improved by making the models predict the semantic representations of the sentences.