Automatic Evaluation of Linguistic Validity in J apanese CCG Treebanks (2025)

Abstract: In natural language inference, the accuracy of systems based on compositional semantics depends on the quality of syntactic analysis, which in turn relies on linguistically valid training and evaluation data, typically provided by treebanks. However, conventional treebank evaluation metrics focus on data coverage and fail to assess the linguistic validity of syntactic structures. This paper proposes novel evaluation methods to enable automatic and multifaceted assessment of linguistic validity. We apply these methods to a Japanese treebank based on combinatory categorial grammar and report the evaluation results.

## introduction natural language inference (nli) is one of the core tasks in natural language processing. among the various approaches to nli, research on inference systems based on compositional semantics has experienced steady research progress @xcite @xcite . in such systems, obtaining linguistically valid syntactic structures and semantic representations is essential because the inference accuracy is strongly influenced by the outputs of syntactic and semantic analyses, which serve as preprocessing for inference. in particular, syntactic and semantic analyses that contain errors can lead to incorrect inference results. specifically, japanese syntactic parsers based on combinatory categorial grammar (ccg) @xcite achieve high accuracy on standard evaluation datasets @xcite , although their outputs have also been reported to lack linguistic validity, especially when handling complex constructions such as passives and causatives @xcite . this discrepancy arises from the fact that both training and evaluation are conducted on treebanks that contain linguistically invalid analyses. this issue fundamentally stems from the absence of established approaches to evaluating the linguistic validity of treebanks. while many treebanks have been developed for various languages and formal grammars @xcite @xcite @xcite , their linguistic validity is yet to be evaluated sufficiently. in many cases, the validity of these resources relies on the assumption that they have been constructed or verified by linguists, but such manual assurances do not offer a quantitative measure of linguistic validity. instead, there is a growing need for principled methods that can evaluate the linguistic validity of treebanks in a systematic way. therefore, this study proposes methods for evaluating the linguistic validity of treebanks from both syntactic and semantic perspectives. we apply these evaluation methods to the japanese ccg treebank constructed by @xcite , and we report our evaluation results. ## conclusion this study proposed syntactic and semantic evaluation metrics for assessing the linguistic validity of treebanks from two independent perspectives. these metrics enable a more fine-grained analysis of structural validity than conventional approaches. ensuring the validity of treebanks is essential not only for improving inference accuracy but also for satisfying requirements such as transparency of error detection and enhanced explainability in future language processing systems. by addressing the lack of principled evaluation methods for linguistic validity, this work offers a step toward more reliable and linguistically grounded approaches in nlp.