Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation (2020)

Abstract: NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU. This position paper argues in favour of two alternative evaluation strategies, using grammars or rule-based systems. These strategies are particularly useful to identify the strengths and weaknesses of different systems. We contrast our proposals with the (extended) WebNLG dataset, which is revealed to have a skewed distribution of predicates. We predict that this distribution affects the quality of the predictions for systems trained on this data. However, this hypothesis can only be thoroughly tested (without any confounds) once we are able to systematically manipulate the skewness of the data, using a rule-based approach.

## introduction recent years have seen many natural language generation (nlg) researchers move away from rule-based systems, and towards neural end-to-end systems. these systems are typically evaluated using textual similarity metrics like bleu @xcite , meteor @xcite , or rouge @xcite , on large corpora of crowd-sourced texts (e.g., the e2e dataset, @xcite @xcite . this evaluation strategy tells us to what extent the generated texts are similar to the reference data, but it is often difficult to determine exactly what that resemblance buys us. by now it is well-known that bleu correlates poorly with human ratings @xcite @xcite @xcite , but bleu by itself also does not tell us anything about the strengths and weaknesses of a particular model, or model architecture. this paper argues that we need alternative (or at least additional) metrics to provide this kind of insight. we believe that rule-based approaches are well-suited for this task. ## systems, models, and architectures before we continue, it is important to recognise the difference between systems, models and architectures. we consider architectures to be abstract descriptions of all the components that make up a system. a system is a specific instance that implements an architecture. when a system is trained on a particular dataset, we can say that it has constructed a model for how the task should be carried out. these distinctions are important, because different nlg researchers may be interested in either systems, models, or architectures. theoretically oriented researchers may be more interested in the properties of different architectures, whereas more applied researchers may be interested in the properties of different systems or models. in our experience, shared tasks are often misconstrued as a competition to see who can deliver the best model. this misses the point, because ideally the results of a shared task teach us about the strengths and weaknesses of different architectures. foot_2 ## evaluating the ability to generalise machine learning datasets are used to determine whether systems are able to generalise from experience to unseen situations @xcite . to test this, researchers typically use separate training, development, and test sets. different models are trained using the training set, the best model is selected using the development set, and then we evaluate its performance on the test set. ## limitations the output of rule-based systems is often said to be less fluent or natural than the output of end-to-end systems, and this claim is corroborated by the results of the e2e-challenge @xcite . it may thus be expected that any synthetically generated corpus will be less natural than humanproduced data, and the texts will probably have other shortcomings, too. however, the proposal in this paper is focused on determining what systems can or cannot learn from corpora with different properties. this means that, to some extent, the naturalness or fluency of the synthetic data does not really matter. what matters is that we learn how those different properties of the data affect the output of data-driven systems. we can then use those systems in other areas, knowing what they are capable of and what their limitations are. at that point, we need a different kind of evaluation (although rules are still valuable to check whether system output conforms to particular guidelines). one might reasonably object here that the quality of the corpus does matter. how can you be sure that your synthetic data has the desired properties? wouldn't that require some form of evaluation as well? we believe this concern could be addressed through unit-tests in the corpus generation code base. because our proposal involves rule-based generation, the output should always be predictable. ## conclusion we discussed the merits of (grammar) rules and rule-based systems in the context of nlg evaluation. our conclusion is that there are clear benefits for practitioners who want to learn more about the architectures that they use for real-life applications. a concern that some may have, is that the real world is messy. why should we solve toy problems like reverse-engineering rule-based systems? our answer is two-fold. first, since our proposals involve synthetic data, we can make the data as clean or messy as we want. but because we have full control over the data, the evaluation will be much more informative about what systems can or cannot do. second, a rule-based perspective is useful because it forces us to engage with the data. looking at the webnlg data, and all of the different templates that exist for each of the different predicates, one cannot help but ask: is this diversity really useful? or should we try to reduce the diversity (e.g. formulating guidelines), to ensure the best possible outputs for our nlg systems? messiness can be good or bad, and it is up to us to explore the impact of variation in nlg data. huiyuan xie, tom sherborne, alexander kuhnle, and ann copestake. 2019. going beneath the surface: evaluating image captioning for grammaticality, truthfulness and diversity. in proceedings of rigorous evaluation of ai systems 2019, collocated with the seventh aaai conference on human computation and crowdsourcing.