Text-in-Context: Token-Level Error Detection for Table-to-Text Generation (2021)

Abstract: We present our Charles-UPF submission for the Shared Task on Evaluating Accuracy in Generated Texts at INLG 2021. Our system can detect the errors automatically using a combination of a rule-based natural language generation (NLG) system and pretrained language models (LMs). We first utilize a rule-based NLG system to generate sentences with facts that can be derived from the input. For each sentence we evaluate, we select a subset of facts which are relevant by measuring semantic similarity to the sentence in question. Finally, we finetune a pretrained language model on annotated data along with the relevant facts for fine-grained error detection. On the test set, we achieve 69% recall and 75% precision with a model trained on a mixture of human-annotated and synthetic data.

## introduction recent neural nlg systems can easily generate fluent texts from linearized structured data @xcite @xcite . however, the systems cannot guarantee that the output is properly grounded in the input -hallucination (outputs not supported by input data) is a notorious problem in neural nlg @xcite @xcite . neural systems are particularly unreliable on complex datasets such as rotowire @xcite , where the task is to generate basketball match summaries from tabular data. rotowire poses multiple challenges for neural systems: it requires content selection and production of longer texts, and its human-written training texts are themselves not always grounded in data, which makes neural models more susceptible to hallucination. on the other hand, rule-based systems used in recent data-to-text tasks @xcite @xcite all achieve high scores in terms of accuracy of the generated contents with respect to the input structures @xcite . this, however, comes with the cost of lower fluency. detecting nlg errors automatically is a hard problem. for word-overlap-based metrics, such as bleu @xcite or meteor @xcite , reliability on content checking is known to be poor @xcite . most neural metrics @xcite have not been evaluated for content preservation. @xcite 's metric based on natural language inference (nli) specifically targets content preservation, but, same as all previously mentioned ones, is not able to provide fine-grained error tagging beyond sentence level. specific content-checking metrics mostly remain a domain of handcrafted pattern matching @xcite , which does not scale well to new domains. while human evaluation provides a more reliable alternative, it is costly and difficult to set up (van der @xcite @xcite . the inlg 2021 accuracy evaluation shared task @xcite aims to improve this situation. @xcite carefully annotated 60 outputs of various neural systems trained on rotowire with 6 error types (see our submission for the shared task falls into the first category: we developed an automatic metric for token-level error annotation which combines a rule-based generation system with a neural retrieval model and a pretrained neural lm used for error tagging. we evaluated our approach in a crossvalidation scenario to select the best configuration for the shared task. overall, our system is able to reach 65% error detection f1 score and ranked first out of four automatic submissions in the shared task. the code for our experiments is freely available on github. foot_0 2 our system our system is composed of 3 steps: a rule-based generator for fact descriptions (see figure 1 and section 2.1), a retrieval system for selecting facts relevant for a given sentence (section 2.2), and a token-level error tagger based on the roberta pretrained lm (section 2.3). the latter two steps are summarized in figure 2 . the lm tagger is trained on examples provided by shared task organizers, as well as on synthetic data based on the rotowire training set (section 2.4). ## experiments we train a pytorch version of roberta from the huggingface transfomers repository @xcite using the adamw optimizer @xcite , learning rate 5 × 10 -5 and linear warmup. we finetune the model for 10 epochs and select the model with the highest validation score. we experiment with several hyperparameters: the results of our model on the development data are listed in ## discussion our charles-upf submission achieved the best results in the automatic metrics category, but there is still a gap with what humans can achieve, as shown by the laval university submission's (garneau and lamontagne, 2021) overall 0.841 recall and 0.879 precision. one way to improve our system would be to enrich the reference fact descriptions, by either inferring more information from the raw data, or by extracting additional data from external databases. 2 another option would be to add surrounding sentences to the context -this could help to resolve coreferences (e.g., if a player is referred to as "he") and to detect the context errors. we also note that our approach requires the real system outputs manually annotated with errors in order to work well -using only synthetic data results in low recall (see ## conclusion we presented our system for detecting errors in generated descriptions of basketball matches. our system can automatically classify the errors on token level, using a pretrained language model and textual description of data generated by a rule-based nlg system. our system reached 0.691 recall and 0.756 precision on the test data, finishing first out of four automatic metric submissions in the inlg 2021 accuracy evaluation shared task.