QCET : An Interactive Taxonomy of Quality Criteria for Comparable and Repeatable Evaluation of NLP Systems (2024)

Abstract: Four years on from two papers (Belz et al., 2020; Howcroft et al., 2020) that first called out the lack of standardisation and comparability in the quality criteria assessed in NLP system evaluations, researchers still use widely differing quality criteria names and definitions, meaning that it continues to be unclear when the same aspect of quality is being assessed in two evaluations. While normalised quality criteria were proposed at the time, the list was unwieldy and using it came with a steep learning curve. In this demo paper, our aim is to address these issues with an interactive taxonomy tool that enables quick perusal and selection of the quality criteria, and provides decision support and examples of use at each node.

## disentangling the what from the how what is being assessed refers to the specific aspect of quality (the qc) that an evaluation aims to assess. how refers to the way in which the qc is mapped to a specific measure that can be implemented in an evaluation method. the distinction matters, because there are many different ways in which the same quality criterion can be assessed @xcite . the different elements relate as follows (evaluation mode: intrinsic vs. extrinsic, subjective vs. objective and absolute vs. relative): quality criterion + evaluation @xmath0; evaluation measure + experimental @xmath1. ## the underlying qc taxonomy the root node is the single most general qc, overall quality. the next level down has the following three qc subclasses @xcite ): 1. correctness: the conditions under which outputs are maximally correct (hence of maximal quality) can be stated. e.g. for grammaticality, outputs are (maximally) correct if they contain no grammatical errors. 2. goodness: it cannot be stated when outputs are maximally good, only which of two is better/worse. e.g. for fluency, even if an output contains no disfluencies, there may be other ways to improve its fluency. 3. features: for a feature-type qc x, outputs are not generally better if they are more (or less) x. depending on context, more x may be better or worse. e.g. funny and entertaining might be good in a narrative generator, but neither are appropriate in a nursing report generator. at the next level, these three nodes split into subclasses that reflect whether outputs are assessed in their own right, relative to the input, or relative to an external frame of reference (e.g. comparison to a gold standard). at a subsequent level, classes further split into the following three subclasses capturing which aspect of an output is being assessed: the rest of the taxonomy consists of 70+ leaf and internal nodes; the following is an example of a complete branch from root to leaf node: overall quality of @xmath2. ## the qcet tool the quality criteria for evaluations taxonomy (qcet) tool has three core functionalities: (i) tree navigation via show child nodes (+) and collapse subtree (-); (ii) tree pruning to one of the subclasses in section 3; (iii) viewing node details. the starting interface (appendix a) has instructions corresponding to section 3 and the use cases below, followed by the three tree pruning pulldowns, and the taxonomy viewer, initially showing just the root node (quality of outputs). the taxonomy tree can then be navigated by progressively showing child nodes and collapsing subtrees. alternatively, to provide less comprehensive views, the taxonomy tree can be pruned in the following three ways: the three pruning dimensions can be combined, so that e.g. selecting correctness qcs only and form qcs only at the same time will show just subtrees that are in both these classes. nodes. each qc node has the following elements: (i) node id, (ii) qc name; (iii) qc definition; (iv) suggested questions to put to the evaluators in a corresponding human evaluation of this criterion (a) in absolute and (b) relative evaluation mode; and (v) additional notes and information. the node id traces the path from the root to the node, e.g. qco-f : @xmath3. each node has a (+) button or a (-) button, standing for show child nodes and collapse subtree, respectively. nodes by default show just the qc name and definition; this view can be expanded in situ, by clicking on show details, or by clicking on the magnifying class icon which opens the full view of the node details in a new tab. see figure 1 for an example of a non-expanded node viewed with details shown. appendix b shows the expanded view of a node. use cases. we envisage two main uses: (i) mapping previous evaluations to standardised qcs for comparability, and (ii) choosing a qc name and definition for new evaluations. in both cases, the default use mode is to peruse the taxonomy from the root node down until the right node is found (more details in appendix c). for users more familiar with the taxonomy, the tree-pruning pull-downs offer a convenient way to reduce the search space. extensibility. the qcet tool is intended as an extensible resource, to which new leaf nodes can be added as needed. the tool includes an interface for proposing new nodes to be inserted at a specific parent node in the publicly shared version which the qcet team will review and, by default, add. ## conclusion we have presented qcet, foot_3 a tool designed to make it easier to (i) identify which standardised qc is being assessed in an existing evaluation, and (ii) to select standard names and definitions to use in new evaluations, in order to increase comparability and repeatability of evaluation experiments overall. qian yu, lidong bing, qiong zhang, wai lam, and luo si. 2020. review-based question generation with adaptive instance transfer and augmentation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 280-290, online. association for computational linguistics.