LLM s Struggle with NLI for Perfect Aspect: A Cross-Linguistic Study in C hinese and J apanese (2025)

Abstract: Unlike English, which uses distinct forms (e.g., had, has, will have) to mark the perfect aspect across tenses, Chinese and Japanese lack sep- arate grammatical forms for tense within the perfect aspect, which complicates Natural Lan- guage Inference (NLI). Focusing on the per- fect aspect in these languages, we construct a linguistically motivated, template-based NLI dataset (1,350 pairs per language). Experi- ments reveal that even advanced LLMs strug- gle with temporal inference, particularly in de- tecting subtle tense and reference-time shifts. These findings highlight model limitations and underscore the need for cross-linguistic evalua- tion in temporal semantics. Our dataset is avail- able at https://github.com/Lujie2001/ CrossNLI.

## introduction recent advances in large language models (llms) have raised important questions about the depth and limits of their language understanding. while these models perform well on many standardized benchmarks, most such evaluations are heavily centered on english and often overlook linguistic features that are specific to other languages. this paper focuses on whether llms have human-like understanding of the perfect aspect of punctual verbs in chinese and japanese. although both languages exhibit features that differ from english (see section 2.1), there has been no systematic investigation of how the perfect aspect is represented or interpreted in these languages within the nli framework. to address this gap, we construct a challenging dataset targeting the interpretation of the perfect aspect with punctual verbs (e.g., die) in chinese and japanese. our dataset is linguistically motivated, template-based, and contains 1,350 sentence pairs per language. our contributions are as follows: ## dataset based on tense (past (pst), present (pres), future (fut)) and the presence (t) or absence (none) of a temporal adverb in the main clause discussed in section 2.1, we designed six japanese sentence templates based on linguistic literature @xcite and created corresponding chinese templates. by using these sentence templates as premises and hypotheses, we constructed 30 premise-hypothesis pairs (p, h) of nli problems for japanese and chinese, respectively. since the perfect aspect with punctual verbs expresses a stable temporal relation in sentences, each (p, h) pair is theoretically expected to have a unique correct label (entailment or non-entailment) under various punctual verb phrases (see a and b in example ( 2 )). this enables us to generate a large number of (p, h) pairs with entailment labels by inserting different lexical items semi-automatically. for 3 months. the examples of sentence templates with labels for chinese are shown in we manually collected 45 sets of common lexical items (nouns and punctual verbs) and clauses to fill our templates. to minimize semantic influence, the items were designed to maintain one-toone semantic correspondence between chinese and japanese. in total, we generated 1,350 (p, h) pairs for each language, comprising 405 instances labeled as entailment and 945 instances labeled as non-entailment. some studies have noted that uncertainty may arise in nli tasks when temporality is involved @xcite . to address this issue, we limited the verb types to punctual verbs that denote irreversible changes (e.g., die). to validate the reliability of the sentences, all instances in the dataset underwent rigorous review and were refined by native speakers. additionally, to ensure labeling reliability, multiple native speakers independently annotated 30 different (p, h) pairs. under a majority voting scheme, their judgments consistently matched the gold labels, demonstrating high inter-annotator agreement. ## results and discussion ## conclusion in this study, we presented a bilingual nli dataset targeting the interpretation of the perfect aspect with punctual verbs in japanese and chinese. our results show that even state-of-the-art llms often fail to capture the correct temporal relations, especially when tense and reference times differ between sentences. our findings highlight the need for evaluation benchmarks that are both linguistically diverse and sensitive to temporal inference.