Jamp: Controlled J apanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models (2023)

Abstract: Natural Language Inference (NLI) tasks involving temporal inference remain challenging for pre-trained language models (LMs). Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages. It is unclear whether current LMs realize the generalization capacity for temporal inference across languages. In this paper, we present Jamp, a Japanese NLI benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. To begin the data annotation process, we create diverse inference templates based on the formal semantics test suites. We then automatically generate diverse NLI examples by using the Japanese case frame dictionary and well-designed templates while controlling the distribution of inference patterns and gold labels. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments (i.e., temporal inference patterns). Our findings demonstrate that LMs struggle with specific linguistic phenomena, such as habituality, indicating that there is potential for the development of more effective NLI models across languages.

## introduction natural language inference (nli) is the task of determining whether a set of premises entail a hypothesis. nli involving temporal inference is a challenging task and remains a significant problem for pre-trained language models (lms). one line of research has investigated the temporal inference abilities of lms @xcite @xcite . however, existing datasets and analyses primarily focus on english, and more analysis and datasets are required for other languages, including japanese. therefore, it is still unclear to what extent current lms can perform various types of means that the gold label is undetermined, → means that the gold label is entailment and ↛ means that the gold label is contradiction. our goal is to construct a temporal inference dataset that precisely assesses the generalization capacities of lms. manual annotation is a viable option for achieving this goal, but it does not fully meet our needs based on several limitations described below. although using crowdsourcing to increase the size of datasets may be cost-effective @xcite , managing biases and artifacts in the resulting data can be challenging @xcite . in contrast, datasets manually constructed by experts @xcite may have high quality but are potentially expensive to scale. additionally, manual dataset construction makes it difficult to control the distribution of vocabulary and inference patterns in a dataset because it heavily relies on the prior knowledge of each annotator (e.g., word choice). to address the issues associated with manual annotation, prior work uses template-based approaches that automatically assign diverse vocabulary to templates that are manually created by experts to construct scalable datasets @xcite . by using this method, we can strictly manage the vocabulary and inference patterns in a dataset, thus it is a suitable approach for probing lms. figure 1 presents our data annotation process, which consists of two stages: template creation and problem generation. we first collect japanese temporal inference examples from jsem @xcite , which is the japanese version of fra-cas @xcite , and manually transform them into templates by masking content words (e.g., nouns and verbs) and temporal expressions (e.g., date and time), producing 46 tense fragments (i.e., temporal inference patterns) based on formal semantics. we then generate examples by assigning content words sampled from a japanese case frame dictionary @xcite and randomly generating temporal expressions to those templates. these techniques ensure that the sentences in jamp are diverse and cover a wide range of temporal inference patterns. it is important to note that our temporal nli examples are derived from a diverse set of templates that are classified with tense fragments, allowing us to create different test splits depending on the goal of evaluation, such as generalization across different tense fragments. we evaluate two japanese models and one multilingual model on our dataset. we analyze whether they can solve our dataset in a zero-shot setting (trained on existing japanese nli datasets) and a fine-tuning setting (trained on a small subset of our dataset). the experimental results demonstrate that the lms can generalize across different tem-poral expressions but fail to generalize some tense fragments such as habituality. ## jamp in this paper, we present jamp, which is a japanese nli dataset for temporal inference, and propose a method for automatic construction from templates based on tense fragments. figure 2 shows the pipeline of our method. first, we create a template by masking content words and temporal expressions in existing temporal nli problems ( §3.1). a template consists of the following triplet: (i) a set of premises in which content words and temporal expressions are masked, (ii) a hypothesis in which content words and temporal expressions are masked, and (iii) a condition for determining a gold label. here, a gold label can take on three values: entailment, contradiction, and neutral. next, we generate training and test sentences by assigning content words selected from the vocabulary list to the template ( §3.2). we create a vocabulary list by using the japanese case frame dictionary to make template p: agent_1 が interval_1 以内に np_1 を vp_1_past 。 h: agent_1 は interval_2 以内に np_1 を vp_1_past 。 g: if @xmath0: エレン が 6 年間 以内 に ゴール を 達成した 。 ellen ga 6 years within ni goal o achieved . (ellen has achieved her goal within six years.) h: エレン は 5 年間 以内 に ゴール を 達成した 。 ellen wa 5 years within ni goal o achieved . (ellen has achieved her goal within five years.) g: neutral ## results and discussion ## conclusion in this study, we constructed jamp, a temporal japanese nli dataset, using a template-based approach. our dataset is controllable in terms of difficulty, vocabulary, and size based on this approach. we conducted experiments using our dataset to probe the generalization ability of pre-trained language models for temporal inference. the experimental results indicated that current lms can generalize for time format splits and time span splits but fail to generalize for tense fragment splits. our dataset demonstrates that there is room for improvement in the generalization ability of current standard lms for temporal inference. because our method is applicable to the construction of datasets for other linguistic phenomena (e.g., modality, comparative), we plan to investigate the generalization ability of language models for other phenomena using the template-based approach in the future.