Shared Task in Evaluating Accuracy: Leveraging Pre-Annotations in the Validation Process (2021)

Abstract: We hereby present our submission to the Shared Task in Evaluating Accuracy at the INLG 2021 Conference. Our evaluation protocol relies on three main components; rules and text classifiers that pre-annotate the dataset, a human annotator that validates the pre-annotations, and a web interface that facilitates this validation. Our submission consists in fact of two submissions; we first analyze solely the performance of the rules and classifiers (pre-annotations), and then the human evaluation aided by the former pre-annotations using the web interface (hybrid). The code for the web interface and the classifiers is publicly available.

## introduction evaluating data-to-text natural language generation (nlg) systems is a very important task despite its notorious difficulty @xcite . reiter and thomson introduced a shared task in evaluating accuracy which consists of factually assessing the accuracy of basketball games summaries produced by an automatic (neural) language generator. the underlying data-to-text dataset was originally created by @xcite evaluating the accuracy of a game's summary relies on identifying the errors within the generated text. borrowing the same terminology as reiter and thomson, the set of possible errors comprises 6 different categories; number, named entity, word, context, not checkable, and other. we refer the reader to the original paper for more details about these different error categories. as part of the shared task, participants were asked to propose an automatic evaluation metric (or algorithm) and/or an evaluation methodology that could be followed by humans in order to assess the accuracy of a given generated text. in our case, we proposed both, i.e. a hybrid approach where the evaluation methodology leverage pre-annotations in order to accelerate (and hopefully improve) the evaluation process. ## evaluating accuracy our evaluation methodology relies on the preannotation of game summaries and a validation procedure that we describe in the following sections. ## discussion as previously mentioned, evaluating the accuracy of generated text in a data-to-text setting is a very hard task. in this experiment, the annotators found the task especially difficult since they are not native english speakers and basketball game descriptions are not (or were not!) a domain with which they were familiar. while the following example may seem trivial, it took several summaries for the annotator to understand that "led the bench" means that it is targeting non-starting players (one could easily assume that every player starts on the bench). also, an annotator who knows which player plays for which team will greatly improve and accelerate the evaluation process. this is something that the annotator just barely began to get comfortable with towards the end of the task. our experiments, due to the lack of time and resources, do not explicitly expose the benefits of having pre-annotations. it would have been interesting to see if, given two annotators with the same prior knowledge foot_3 , the pre-annotations helps them to solve the task both in a matter of time and accuracy, hence evaluating the inter-annotator agreement. nonetheless, the pre-annotations did help the annotators through the evaluation process in three different ways; 1. get familiar with the task, domain, and dataset by seeing which potential word may be erroneous; we decided to design our own annotation tool instead of using webanno @xcite , the one initially used by the authors of the task. this decision was mainly motivated by the flexibility of storing and pre-annotating the game summaries. in future works, we would consider more robust text classifiers coupled with active learning in order to improve the pre-annotations, while executing the task. we would also consider adding contextual information from the box score for the classification.