Decomposed scoring of CCG dependencies (2023)

Abstract: In statistical parsing with CCG, the standard evaluation method is based on predicate-argument structure and evaluates dependencies labelled in part by lexical categories. When a predicate has multiple argument slots that can be filled, the same lexical category is used for the label of multiple dependencies. In this paper, we show that this evaluation can result in disproportionate penalization of supertagging errors and obfuscate the truly erroneous dependencies. Enabled by the compositional nature of CCG lexical categories, we propose *decomposed scoring* based on subcategorial labels to address this. To evaluate our scoring method, we engage fellow categorial grammar researchers in two English-language judgement tasks: (1) directly ranking the outputs of the standard and experimental scoring methods; and (2) determining which of two sentences has the better parse in cases where the two scoring methods disagree on their ranks. Overall, the judges prefer decomposed scoring in each task; but there is substantial disagreement among the judges in 24% of the given cases, pointing to potential issues with parser evaluations in general.

## introduction with a suitably designed architecture, combinatory categorial grammar (ccg) supertaggers can learn to better maintain syntagmatic consistency, adjusting for their own errors to keep the sentence parsable @xcite . this kind of adjustment, however, comes at the expense of its evaluated word accuracy, which is the prevailing evaluation measure for supertagging. the standard, final parser evaluation is no kinder in such cases: it examines induced bilexical dependencies, but these are labelled (in part) by the lexical category assigned to the head word @xcite the code for decomposed ccg scoring is available online at https://www.cs.toronto.edu/~aditya/ccgds @xcite . if the category is incorrect, its outgoing dependencies are considered incorrect. while other areas of nlp such as natural language generation and machine translation have recently warmed to efforts to validate their intrinsic evaluations against human judgements (e.g., @xcite , this has not been the case so far with statistical parsing. this is likely because evaluating the quality of a syntactic parse requires grammatical expertise. in this paper, we examine ccg parser evaluation and identify a number of cases where the standard ccg scoring method gives undesirable results. we address these shortcomings by introducing decomposed scoring. to evaluate our new method, we elicit judgements from categorial grammar (cg) experts in two pairwise selection tasks using englishlanguage data from ccgbank @xcite . in the first, intrinsic task, the judges are directly asked which of the two scoring methods they prefer for a given sentence. we find that they prefer decomposed scoring in 90% of the cases presented. in the second, extrinsic task, judges are given two different sentences, each with an erroneous parse, where the scoring methods disagree about which parse should be ranked higher. here, we find that the judges do not have majority agreement in 24% of the cases presented; but where they do reach majority consensus, they agree with decomposed scoring in 62% of cases. the high disagreement raises important questions about statistical parser evaluations, which we discuss. ## background following @xcite , the standard evaluation measure for ccgbank-based ccg parsers is f₁ over bilexical dependencies. each dependency represents a predicate-argument relationship as indicated by the corresponding lexical categories. a (labelled) ccg dependency 𝑑 is defined as a 4-tuple 𝑑 = (ℎ p , ℎ a , c, 𝑠) where: given an input sentence, the set of corresponding ground-truth dependencies 𝒟 g from ccgbank, and a candidate set of dependencies 𝒟 c from a parser, the candidate dependencies are evaluated according to the f₁ score between the two sets: this dependency-based measure directly evaluates the parser's ability to produce the intended predicate-argument structure. models that analyze sentences with different derivations than the one provided by the corpus will not be penalized unless the derivation alters the semantics-i.e., this evaluation is invariant to spurious ambiguities. ## decomposed dependency scoring in this paper, we do not take issue with the treatment of the head words ℎ p and ℎ a , but rather of the lexical category, 𝑐, and argument slot number, 𝑠. we alter how ccg dependency labels are compared so that a dependency's correctness no longer requires the entire lexical category to be correct and allow (judicious) flexibility in valid values for the slot number. as these modifications are dependent on subcategorial decompositions of the lexical category labels, we term the overall approach decomposed scoring. ## evaluating decomposed scoring thus far, we have argued for decomposed scoring on the basis of examples that show cases where decomposed scoring is able to more precisely isolate and penalize parsing errors. our next aim is to determine the extent to which this capability holds true over a larger set of parser outputs when evaluated by expert judges in a systematic manner. from here, we refer to the standard ccg evaluation as f₁ and to decomposed scoring as df₁. ## results and discussion in the first task, the judges ruled strongly in favour of df₁, agreeing with it in ¹⁸⁄∕ 20 cases. this is statistically significant (binomial test, 𝑝 ≈ 2.0 × 10 -4 ). we therefore conclude that df₁ is better than f₁ at identifying the ultimate error in the parser's output. in the second task, we find some disagreement among judges: for ¹¹⁄∕ 45 sentence pairs, two judges agreed with df₁, while the remaining two disagreed with it, leading to a tie. but of the remaining 34 sentence pairs, a majority of judges agreed with df₁ on 21 pairs. even with the ties, this is also statistically significant (𝑝 ≈ 0.02); we thus conclude that df₁ is preferable to f₁. disagreement among judges in the second task merits discussion. judges are taken to be providing ground truth, so tied pairs are cases where a ground truth judgement is unavailable. what does this mean for decomposed scoring, and for ccg parser evaluations more generally? concerning decomposed scoring, the first task's results empirically validate the utility of df₁ over f₁. df₁ prevents obfuscation of erroneous dependencies, improving the granularity of the evaluation measure. in addition, there is the possibility of helping with the training of statistical parsers: when a dependency's only crime is sharing a lexical category with an erroneous one, training a parser to learn that both are errors may cause it to learn to avoid correct dependencies. moreover, we examined the sentence pairs in the second task ourselves and found that even when we judged df₁'s rank inversion to be incorrect, undoing df₁'s changes would not address the issue; the severity of parsing errors varies and is in part modulated by semantic intricacies. we thus expect that the disagreements among judges are due at least in part to underlying differences in opinions about sentence meaning and/or salience. turning to the broader issue of ccg parser evaluations, the lack of definitive ground truth for many inter-sentence comparisons implies limitations for inter-parser comparisons. imagine a case where parser a claims to outperform parser b, and closer inspection reveals that a and b differ in their outputs on only two sentences. for one of these sentences, a's output has a higher f₁ than does b's; for the other, the opposite is true, but the difference in f₁ is smaller. now, the claim that a outperforms b becomes a claim that the parse that a produces for the first sentence is better than the parse that b produces for the second. and yet, as indicated by the judges' disagreements in the second task, it is not always possible to make these kinds of judgements. ## conclusion we have found that the standard ccg evaluation method's choice of dependency labels is prone to amplifying minor errors. we proposed decomposed scoring and validated it by consulting experts. from their judgements, we conclude that decomposed scoring is better at isolating parser errors and is, overall, a better choice than the standard scoring method. disagreement in the second task, however, is a source of concern and suggests potential issues for ccg parser evaluations. given the frequentlysmall deltas between modern parsers, this is worth investigating further.