Reforging : A Method for Constructing a Linguistically Valid J apanese CCG Treebank (2024)

Abstract: The linguistic validity of Combinatory Categorial Grammar (CCG) parsing results relies heavily on treebanks for training and evaluation, so the treebank construction is crucial. Yet the current Japanese CCG treebank is known to have inaccuracies in its analyses of Japanese syntactic structures, including passive and causative constructions. While ABCTreebank, a treebank for ABC grammar, has been made to improve the analysis, particularly of argument structures, it lacks the detailed syntactic features required for Japanese CCG. In contrast, the Japanese CCG parser, lightblue, efficiently provides detailed syntactic features, but it does not accurately capture argument structures. We propose a method to generate a linguistically valid Japanese CCG treebank with detailed information by combining the strengths of ABCTreebank and lightblue. We develop an algorithm that filters lightblue’s lexical items using ABCTreebank, effectively converting lightblue output into a linguistically valid CCG treebank. To evaluate our treebank, we manually evaluate CCG syntactic structures and semantic representations and analyze conversion rates.

## introduction there have been significant advances in natural language processing research through the construction of syntactic tree corpora, known as treebanks. treebanks are datasets where syntactic structures are annotated over large bodies of text. various treebanks @xcite @xcite @xcite @xcite have been served as standard datasets for training and evaluating statistical syntactic parsers. the penn treebank @xcite , one of the first context-free grammar (cfg) treebanks, contains a one-million-word corpus of wall street journal text. ccgbank @xcite was constructed by converting the penn treebank to combinatory categorial grammar @xcite @xcite , which contributing to the advancement of ccg parsers. there are various methods for constructing treebanks. one approach is to combine automatic part-of-speech (pos) taggers and syntactic parsers with manual corrections, as in the penn treebank. the approach to providing ccgbank involves automatic conversion from existing treebanks. however, treebanks have different formats, provided information, and informational validity. japanese ccgbank @xcite , constructed by automatic conversion of japansese dependency tree corpora, but provides limited validity of syntactic structures for passive or causative nestings @xcite . the japanese abc grammar treebank abctreebank @xcite has manually annotated argument structures, but does not provide pos information (conjugation series, conjugation forms, among others) and other detailed information. we thus aim to construct a japanese ccg treebank with both linguistically valid syntactic structures and detailed syntactic features. to this end, we propose a method to construct a new japanese ccg treebank using the japanese ccg parser lightblue @xcite 1 , which can output detailed syntactic features. however, lightblue contains errors related to argument structures, causing inaccurate outputs. to address this drawback, we extracted predicate-argument structures from abctreebank and incorporated this information into lightblue. this approach involves decomposing and reconstructing the treebank, which we call "reforging". we discuss the reforging process in more detail in section 3. section 4 assess the validity of our proposed method. section 5 introduces the error analysis of the output trees. section 6 describes manual evaluations considering the syntactic structures and semantic representations of the output trees. in ccg, lexicons associate words with their phonological and syntactic information. for instance, to analyze the sentence (1) taro runs. in ccg, the the lexical items t aro : n p runs : s\n p are supposed to be contained in the lexicon. combinatory rules allow syntactic categories to be merged. function application rules and function composition rules are the basic ccg rules, defined as follows: the ccg syntactic structure of the sentence (1) is given using the following function application rule: in ccg, a combinatory rule applied to syntactic structures is indicated by a symbol placed on the right of the horizontal line. for example, in the above syntactic structure, the symbol "<" on the right of the horizontal line indicates the use of the backward application rule. ccg also includes other rules such as coordination, crossedcomposition, and crossed-substitution rules. @xcite constructed japanese ccgbank through automatic conversion of the kyoto corpus foot_1 , naist text corpus foot_2 , and japanese particle corpus @xcite . however, japanese ccgbank has some empirical problems. one of the problems, discussed by bekki and yanaka @xcite , is that the syntactic analysis of the japanese ccgbank contains empirical fallacies on predictions for passive and causative nestings. ## the reforging process the aim of this research is to construct a linguistically valid japanese ccg treebank with detailed syntactic features. therefore, we propose a method for constructing a japanese ccg treebank by combining the positive aspects of abctreebank, in which argument structures are manually annotated, with lightblue's ability to provide ccg trees with detailed syntactic features. the proposed method decomposes abctreebank and reconstructs it using lightblue, a method we call "reforging". note that "reforging" is not a commonly used linguistics term, but decomposing and reconstructing a treebank using a parser is a novel approach, so we assigned this name for convenience. one approach to constructing a new ccg treebank would be to automatically correct japanese ccgbank output. however, one difference between ccg and cfg is that modifying part of a ccg tree would require recalculating all subsequent calculations. since those recalculations would be almost as costly as reparsing the entire sentence, this study aims to construct a treebank using reforging. the reforging process has three steps: figure 3 shows each process in detail. a parser decomposes a given abctreebank input and its lightblue reconstruction is output as a treebank. ## discussion figure 6 shows the syntactic structure tree of the sentence (3) before filtering the lightblue chart, and figure 7 shows the tree after filtering. the lightblue lexical items for the predicate gather are the entries having ga-case and ni-case n p s as arguments. therefore, gather was analyzed as having not only ga-case n p but also ni-case n p as arguments using the chart before filtering. however, the verb gather does not necessarily have the ni-case n p as an argument. especially in sentence (3), it is reasonable to assume that gather has only ga-case n p as the argument. by overwriting the partial syntactic structure of the predicate gather having ga-case and ni-case n p s as arguments with a lexical entry having only gacase n p as the argument, it became possible to convert abctreebank sentences to linguistically valid ccg syntactic structures. lightblue output also contains detailed syntactic features and dependent type semantics (dts; @xcite representations. ## error analysis the reforging process is successful in some sentences, but errors can still occur due to factors such as incorrect argument structures in abctreebank or incorrect analysis of the adnominal clause in lightblue. ## evaluation since the linguistic validity of constructed treebanks cannot be automatically evaluated, it is necessary to manually check one-by-one whether each syntactic structure and semantic representation is correctly obtained. however, the cost of manually evaluating the syntactic structures of ccg and dts representations would be very high, and it was unfeasible to manually evaluate all of the 13,653 sentences constructed in this study. we thus manually evaluated 56 sentences, randomly sampling four sentences from each genre. we also evaluated the constructed treebanks by their conversion rates. as evaluation metrics for machine-learning-based ccg parsers such as depccg @xcite , lexical coverage, sentential coverage, and syntactic rule coverage are used. however, we did not use supervised learning methods for ccg parsing; instead, we performed rule-based conversions. consequently, evaluations using unseen data were infeasible, so we used only conversion rates as an evaluation metric. ## conclusion we proposed a reforging method for constructing linguistically valid japanese ccg treebanks with detailed syntactic features. our method obtained correct japanese ccg syntactic structures to some extent. our method assumes that abctreebank argument structures are valid because their syntactic structures are manually annotated. however, there is an upper bound on the validity of abctreebank argument structures. to obtain linguistically valid argument structures, our future work will consider combining abctreebank with other reliable resources. we also plan to improve our filtering algorithm, and improve lightblue's parsing algorithm to better handle long sentences. figure 10: ccg tree of the passive sentence constructed by lightblue