Standard Quality Criteria Derived from Current NLP Evaluations for Guiding Evaluation Design and Grounding Comparability and AI Compliance Assessments (2025)

Abstract: Research shows that two evaluation experiments reporting results for the same quality criterion name (e.g. Fluency) do not necessarily evaluate the same aspect of quality. Not knowing when two evaluations are comparable in this sense means we currently lack the ability to draw conclusions based on multiple independently conducted evaluations. It is hard to see how this issue can be fully addressed other than by the creation of a standard set of quality criterion names and definitions that the evaluations in use in NLP can be grounded in. Taking a descriptivist approach, the QCET Quality Criteria for Evaluation Taxonomy derives a standard set of 114 quality criterion names and definitions from three surveys of a combined total of 933 evaluation experiments in NLP, and structures them into a reference taxonomy. We present QCET and its uses in (i) establishing comparability of existing evaluations, (ii) guiding the design of new evaluations, and (iii) assessing regulation compliance.

## introduction natural language processing (nlp) uses a wide variety of different names to refer to what is assessed in system evaluations, estimated to number over 200 for text-generating systems alone @xcite . details of evaluations and definitions of what they assess are mostly patchy @xcite @xcite , and it is often impossible to tell if the same aspect of quality was assessed in two evaluations, resulting in unclear comparability and low repeatability @xcite . consider the following evaluations which use the same name, fluency, but each with a different definition: the first two evaluations assess single but different criteria (utterances can be grammatical but not very fluent), the third assesses two (an utterance can be grammatical yet illogical), so none of them assess the same aspect of quality. it is common for what was actually evaluated to be at odds with the name (or even definition) given in papers @xcite , e.g. the second evaluation above claims to be evaluating fluency, but actually evaluates grammaticality. in this situation, not only is it misleading to report, say, that a system improves fluency, but any comparisons with other fluency assessments are also unsafe. as has been argued @xcite @xcite @xcite , this is problematic, in particular for human evaluation which has always been considered the litmus test of quality in nlp. it is hard to see how the current misalignments between (i) what is actually evaluated vs. what it is named, and (ii) what different researchers mean by the same quality criterion name, can be addressed other than by a standard reference set of criterion names and definitions that those actually in use can be grounded in. deriving such a resource from reported evaluations has been our aim in the work reported here, resulting in the qcet quality criteria for evaluation taxonomy, consisting of the following resources (browser at https://nlp-qcet.github.io/ ; other items at https://github.com/dcu-nlg/qcet_code ): ## standardising quality criteria qcet is based on the notion of quality criterion (qc), i.e. the specific aspect of system quality that is assessed in an evaluation, and the level at which we would expect two (well-designed) evaluations (that assess the same qc) to support the same conclusions about which of two systems is better. consider these descriptions of two evaluations: q: what does this evaluation consider a better system to be? in the above evaluations, a better system is not one that is found to be better in an experiment with 8 users that access an airline schedule database to book a specific flight while talking to the system and take less time to do it. user numbers, interaction mode, location of information, etc., are not part of the aspect of system quality assessed, but of experiment design or system implementation. instead, once we disregard such factors, we are left with the following answer to the above question: a: a better system is one that enables the user to complete a given task more quickly. at this level we can see that both evaluations above assess the same aspect of quality. we would expect two well-designed evaluations that assess the same pair of systems in terms of this aspect of quality, using the same task and data, to come to the same conclusion about which is better. 1 the above process is in fact how we derive standard qc definitions from three surveys of nlp evaluations. the first @xcite proposed 71 standardised qcs for human evaluation of text-generating systems which we reviewed and revised for inclusion in qcet. we conducted two new surveys (belz et al., to appear) of 60 papers each, published in 2022-2024 in acl main proceedings. here our scope was all of nlp, not just nlg as was the case in the 20years survey. we identified altogether 455 individual occurrences of qcs, among which were 19 new ones which we added to the taxonomy. based on the three surveys we ended up with 114 qcs for qcet. for qc names, we follow the convention that we name qcs after the 'good end' of the correspond-ing quality spectrum, or both ends where there isn't one (see section 3), also aiming to closely reflect the definition. e.g. the qc assessed in the above evaluations is task completion speed. as a taxonomy of quality criteria, qcet is agnostic about, and therefore covers equally, automatic and human evaluations, all types of nlp systems and output modalities @xcite , with the exception of certain individual terminal nodes (e.g. spelling accuracy is defined only for textual output). ## conclusion we have presented qcet, a taxonomy of 114 standard quality criteria complete with definitions and attestations from the literature, derived from a combined total of 933 existing nlp evaluation experiments. qcet is designed to support (i) assessments of the comparability of existing evaluations, (ii) guiding the design of new evaluations that are comparable by design, and (iii) assessment of regulation compliance. rather than attempting to achieve complete coverage of qcs currently in use, qcet is designed to be extensible by adding new qcs to the appropriate branches of the taxonomy.