A proof of concept on triangular test evaluation for Natural Language Generation (2020)

Abstract: The evaluation of Natural Language Generation (NLG) systems has recently aroused much interest in the research community, since it should address several challenging aspects, such as readability of the generated texts, adequacy to the user within a particular context and moment and linguistic quality-related issues (e.g., correctness, coherence, understandability), among others. In this paper, we propose a novel technique for evaluating NLG systems that is inspired on the triangular test used in the field of sensory analysis. This technique allows us to compare two texts generated by different subjects and to i) determine whether statistically significant differences are detected between them when evaluated by humans and ii) quantify to what extent the number of evaluators plays an important role in the sensitivity of the results. As a proof of concept, we apply this evaluation technique in a real use case in the field of meteorology, showing the advantages and disadvantages of our proposal.

## introduction evaluation can be defined as "the systematic determination of the merit, value and meaning of something or someone based on criteria with reference to a set of rules" @xcite . for some authors, the concept of evaluation appeared in the 19 th century with the industrialization process in the u.s. @xcite . later on, a modern scientific discourse emerged in the field of education that would incorporate terms such as learning objectives or educational assessment @xcite . nowadays, evaluation has been extrapolated to many areas beyond education and consists of the process of obtaining evidence that allows to judge the degree of achievement of previously established objectives. nevertheless, despite the technological advances in recent years, there are still certain areas in which the evaluation process must be carried out by humans and not just based in data-driven metrics. in these cases, it is difficult to avoid subjective judgments in the evaluation process. evaluation of an nlg system usually requires checking the degree to which it meets the established language requirements, such as the quality of the texts generated, their correction, their interpretability, syntax, formatting or style. the task of evaluating nlg systems presents difficulties mainly because usually these systems do not produce a single correct output and therefore it is hard to define universally accepted metrics for nlg evaluation. when conducting an nlg evaluation with users, there is no general consensus about what to ask (e.g., "how fluent do you think the text is?" or "how natural do you think the text is?"), how many evaluators should participate in the assessment process, or which specific statistical tests should be applied. moreover, subjectivity can influence the evaluation results and make them be devoid of statistical significance. although some authors have advised against the use of statistical significance testing in corpus linguistics @xcite , there have been several proposals for addressing the effect of human subjectivity and statistical significance in human evaluation for several computational linguistics related tasks. in this regard (van der @xcite presents an overview of statistical significance tests that are conducted in human evaluation in nlg. they summarize also a set of best practices grounded in the literature. in addition, @xcite describes a survey of methods for measuring agreement among corpus annotators. moreover, @xcite shows the limits of considering inter-annotator agreement as the only criterion for checking evaluation reliability, and proposes correlation coefficients and agreement coefficients to be used together with the aim of obtaining a better assessment of the data reliability for human evaluation in nlg. in spite of this, and to the best of our knowledge, in the evaluation of nlg systems, so far, there are no established protocols or standards to successfully minimize the effect of human subjectivity and to ensure that results are reported with statistical significance as exist in other areas, such as for example, sensory analysis. in this realm there are well established procedures and rules for the humanbased measurement of the sensory characteristics of products @xcite ; european sensory science society, 2020) that guarantee the validity of the evaluation results and their statistical significance. sensory analysis and the computational theory of perceptions have already been applied to automatic reporting @xcite . in this paper, we propose a technique for nlg evaluation that is supported by some of the standards applied in sensory analysis. it consists of a manual evaluation that allows to obtain a global assessment of the generated texts, instead of assessing a unique characteristic (e.g., fluency or coherence). as a matter of fact, the new technique minimizes the subjectivity inherent to human evaluation. we also present the experimental results obtained when carrying out a proof of concept of this technique by comparing real texts generated by two different people. the objective of this preliminary experimental study is to analyze in practice the advantages and disadvantages of the proposed technique before applying it to the evaluation of an end-to-end nlg system that is currently under development. the rest of the manuscript is organized as follows. in section 2 we provide a summary of the state of the art of nlg evaluation techniques. in section 3 we introduce some preliminary concepts in the field of sensory analysis needed to understand the new evaluation technique proposed in section 4. in section 5 we present the experimental setting and reported results. finally, section 6 concludes with final remarks and possible future work. ## background evaluation of nlg systems is very different from other areas, because of the number and type of different dimensions to be considered. accordingly, it constitutes one of the current challenges in research in the nlg field. in the review by van der @xcite , the authors present a review of the challenges of evaluating nlg systems, the pros and cons of different evaluation approaches, and a guide to good practice in conducting nlg evaluation. they emphasize the need to conduct an evaluation with people (whenever possible), in addition to using several independent evaluation criteria. they also recommend that the number of evaluators required be duly justified, as well as their socio-demographic profile, and preferably that the evaluation panel be designed with the widest possible audience in mind. the random and balanced design of the samples to be evaluated, as well as their number and order, should also be justified in order to minimize possible biases and the subjectivity of results. finally, regarding statistical analysis, it is proposed to distinguish between exploratory studies (more qualitative) and confirmatory studies (more quantitative and supported by results with statistical significance). in addition, when evaluating an nlg system, it may be necessary to consider aspects related to the final text generated by the whole system or specific aspects of one or several stages of the generation process (e.g., content determination, lexicalization, surface realization, etc.). due to the great number and diversity of characteristics to be considered when evaluating an nlg system (e.g., readability of the texts generated, coherence, interpretability, etc.), different strategies can be followed @xcite : extrinsic versus intrinsic evaluation. extrinsic evaluation deals with assessing the impact of the system on users or other tasks, focusing on the effects produced by the system (e.g., assessing the decisions made by users based on the system output). intrinsic evaluation (e.g., evaluating the degree of fluidity of the texts generated) pays attention to the effectiveness of the system itself. in addition, we need to distinguish between automatic and manual evaluation @xcite . the former is based on metrics that automatically compare the system-generated text with a human-generated text corpus, while the latter requires the participation of humans. it is worth mentioning that a key issue comes into play in human evaluation: how to handle the subjectivity introduced by the evaluators when judging the system-generated text. it is quite common for intrinsic evaluation to be carried out using automatic metrics such as bleu or rouge @xcite and for extrinsic evaluation to be carried out manually. however, in some cases more than one type of evaluation needs to be considered because they are complementary @xcite . when it comes to selecting the evaluation technique, there are a multitude of automatic and manual techniques that can be applied, but unfortunately there is not still an adequate cataloguing and characterization of them. generally, manual evaluations tend to be more costly in both time and money. a clear example of manual evaluation that involved considerably high costs (20 months and 75,000gbp) was the evaluation of the stop system @xcite , which automatically generated personalized letters to encourage users to stop smoking. on the other hand, automatic type metrics are usually cheaper and allow for quick results. however, some aspects of a text generated by an nlg system, such as correctness or consistency, are difficult to evaluate by automatic metrics. in these cases, a manual evaluation is most appropriate, where, usually, evaluators are asked to rate or rank several texts @xcite . in other cases, a task-based evaluation is also carried out, whereby evaluators must make a decision based on the output provided by the system @xcite . automatic metrics assess the degree of accuracy or objectively score how good the output of a system is with respect to the evaluated issues. however, when it comes to human evaluation, the main problem is the inherent subjectivity of each evaluator. therefore, the introduction of standards and or protocols for obtaining objective and statistically significant results in the context of human evaluation would be highly appreciated by the nlg community. ## preliminaries sensory analysis is a well-established scientific discipline with a wide range of applications (e.g., tasting cheeses, oils, wines, creams, etc.) and standards for human evaluation, which are developed by the international organization for standardization (iso) @xcite . in this paper we adapt an evaluation technique from sensory analysis to nlg. in order to understand our proposal, basic concepts of sensory analysis are introduced below: the usual values of α in the field of sensory analysis and those we will use in this paper are {0.001; 0.01; 0.05; 0.1; 0.2} depending on the sensitivity required by the test. ## the nlg triangle test the evaluation technique proposed in this paper consists of a triangle test taken from the sensory analysis research field and adapted to nlg evaluation. thus, instead of presenting the judges triads of food samples in which two of them are the same and one is different, they will be shown three text samples, two generated by the same subject and a third generated by a different subject. in this way, the judges will have to identify which one of the text samples in the triad has been written by a different subject from the other two. it is worth noting that this technique is applicable regardless how the texts under consideration were generated, either manually by humans or automatically by nlg systems, i.e., no matter if each subject is a human or an nlg system. ## use case with the aim of developing a proof of concept of the technique presented in the previous section, we have applied the nlg triangle test to texts in the meteorological field, an area in which we had designed an nlg system previously @xcite . however, it is worth noting that the texts generated in @xcite are weather forecasts by the local council and do not take into account the whole region. for the sake of simplicity in the recruiting of judges, our use case deals with texts which describe the weather forecast for the entire region and are written by meteorologists. for illustrative purpose, we considered expert judges (see section 5.1) and non-expert judges (see section 5.2). judges were asked to fill in a questionnaire @xcite which is divided into several questions. in each question (see fig. 1 ), three meteorological situations were presented. each situation consisted of an image showing the state of the sky for one day in galicia and a short text written by a meteorologist from the meteorological observation and prediction unit of the galician meteorological agency (meteogalicia foot_5 ). of the three situations presented in each question, there were two in which the descriptive text had been created by the same subject and a third in which the creator was a different subject. the judge had to select the text that he/she believed to be created by a different subject from the one who had written the other two. ## final remarks and future work we have proposed in this paper a new technique for the evaluation of nlg systems. this technique allows us to obtain statistically significant results with the least possible subjectivity from an evaluation carried out by humans, either experts or nonspecialists. our technique provides a mechanism to compare two texts generated by different subjects (either humans or machines) and determines whether difference is detected between them or not. in the given illustrative use cases, we have learned a number of lessons regarding the type of test. in case of a test of difference, repeated evaluation by judges is allowed. therefore, each judge can perform several evaluations and be treated as independent. however, in the similarity test repeated assessments are not allowed. therefore, to obtain equivalent sensitivity levels in a test of difference and in a similarity test, approximately twice as many judges are needed in the similarity test. we have also seen that and quantified to what extent the number of judges plays an important role in the sensitivity of the results. although the guidelines in section 4.1 indicate that first the sensitivity values must be determined and then the number of judges, in practice, it is likely that an unlimited number of judges with the required profile will not be available for the evaluation, and therefore sensitivity values will be decided based on the number of judges available. therefore, if high sensitivity values are required, then a large number of judges must be available. in any case, if the sensitivity level is imposed a priori by the case of study, it will determine the minimum number of judges needed to perform the nlg triangle test. as the number of evaluators increases, the degree of confidence in the test results also increases. however, if a specific profile of evaluators is required and their availability is low, even with a not very large number of evaluators it is possible to obtain results with a confidence level that in many cases exceeds 90%. in this case, quantitative evidence would support the quality of the texts produced. if a large enough number of evaluators, confidence levels close to 100% can be achieved applying a triangle similarity test. in this case, the conclusion is that empirical evidence shows that a large part of the population does not detect difference between system-generated texts and humangenerated texts. achieving results in this range may require a very large number of evaluators, which in many practical contexts would make the test unfeasible. as future work, we will apply our nlg triangle test to the comparison between texts generated by nlg systems and texts generated by humans. we will also aim to extend the evaluation technique by including mechanisms for allowing judges to express their motivation for the answers provided and take into account this additional information in the analysis of results.