TGEA : An Error-Annotated Dataset and Benchmark Tasks for T ext G eneration from Pretrained Language Models (2021)

Abstract: In order to deeply understand the capability of pretrained language models in text generation and conduct a diagnostic evaluation, we propose TGEA, an error-annotated dataset with multiple benchmark tasks for text generation from pretrained language models (PLMs). We use carefully selected prompt words to guide GPT-2 to generate candidate sentences, from which we select 47K for error annotation. Crowdsourced workers manually check each of these sentences and detect 12k erroneous sentences. We create an error taxonomy to cover 24 types of errors occurring in these erroneous sentences according to the nature of errors with respect to linguistics and knowledge (e.g., common sense). For each erroneous span in PLM-generated sentences, we also detect another span that is closely associated with it. Each error is hence manually labeled with comprehensive annotations, including the span of the error, the associated span, minimal correction to the error, the type of the error, and rationale behind the error. Apart from the fully annotated dataset, we also present a detailed description of the data collection procedure, statistics and analysis of the dataset. This is the first dataset with comprehensive annotations for PLM-generated texts, which facilitates the diagnostic evaluation of PLM-based text generation. Furthermore, we use TGEA as a benchmark dataset and propose a series of automatic diagnosis tasks, including error detection, error type classification, associated span detection, error rationale generation, to further promote future study on the automatic error detection and correction on texts generated by pretrained language models.

## introduction pretrained language models @xcite @xcite , which are trained on a huge amount of data via self-supervised learning, have made remarkable progress on both natural language understanding (nlu) @xcite and natural language generation (nlg) @xcite @xcite . on several nlu datasets, plm-based neural models have gradually achieved human-level performance in terms of automatic evaluation metrics (e.g., accuracy, f 1 ) @xcite . in order to deeply understand and analyze the capability of plms on nlu, a variety of more challenging nlu datasets have been proposed @xcite @xcite . these datasets can be used not only to obtain knowledge on how plm-based models work and what they learn, but also to define new nlu tasks and to serve as a benchmark for future progress. for example, evaluating and analyzing plm-based models on learning document structures with a carefully created benchmark test suite @xcite , helps to develop new methods to enhance the capability of these models on discourse modeling @xcite . knowing the weakness of current plm-based models in commonsense reasoning @xcite has inspired people to develop various reasoning datasets @xcite . on the other hand, state-of-the-art plms are able to generate texts that are even not distinguishable from human-written texts by human evaluators @xcite . this makes us curious about the capability of plms on text generation. are they really reaching humanlevel performance on text generation? in contrast to the studies of plms on nlu, research on the capability of plms on nlg is quite limited, especially in dataset building and diagnostic evaluation of text generation errors. in this paper, in order to recognize the perimeter of text generation capability of plms, we propose tgea, an error-annotated dataset with multiple benchmark tasks for text generation from pretrained language models. the original raw data are collected from texts generated by a chinese gpt-2 model. the entire data collection and annotation procedure is visualized in figure 1 . the goals and contributions of building tgea are as follows. t p : c ec i c ec ④ e e c a ifica i ② e e a d a cia ed s a de ec i ③ e c ec i ⑤ ra i a ge e a i c gpt-2 别不同。 sho p and archer are of differen ca egorie . 动员 中 冠军。 japanese archery players won the championship in women's shot put competition. 动员 中 冠军。 japanese archery players won the championship in women's shot put competition. 动员 中 冠军。 japanese shot putters won the championship in women's shot put competition. japan 别不匹 category mismatch te ge e a i ① e e te de ec i a very different error distribution, especially on errors related to commonsense reasoning. our work is related to gec datasets in error annotation and correction (machine vs. human errors). it is also partially related to commonsense reasoning datasets that have been proposed recently in that our dataset includes commonsense reasoning errors and rationales behind these errors. our dataset is not related to conventional text generation datasets @xcite @xcite for training text generation models. a comprehensive comparison to gec datasets and commonsense reasoning datasets is shown in ## tgea as a benchmark we use our dataset as a benchmark and propose 5 tasks that are defined for errors in texts generated by plms. we provide baseline results for these tasks in this section. we employ three bert-style chinese plms as baselines in our experiments, namely bert-wwmext, roberta-wwm-ext-large developed by @xcite 5 and albert-chinese-large 6 . for notational simiplicity, we denote them as bert zh , roberta zh and albert zh respectively. please refer to the appendix for the model hyperparameter settings of each task. ## discussion since we use machine-generated texts for error annotation, hyperparameters of models (e.g., sampling strategies, model size), model types (e.g., gpt-2, gpt-3 or other plms for text generation), and genres of texts used to train plms, etc., all have impacts on generated texts and hence on error types and error distribution. a straightforward way to mitigate this issue is to collect raw texts from multiple models with different hyperparameters, neural architectures and text genres. this will lead to an expanded dataset with a much larger number of instances to be manually annotated, which is expensive and time-consuming. yet another issue with this is that it may result in a bunch of data due to inconsistency across different models and difficulty in setting the proportion of each data source. instead, we focus on consistently annotating errors for texts generated from a single source. in order to make tgea as general and representative as possible, we use gpt-2 that is not only currently state of the art in text generation but also easily available. we also adopt standard and widely-used hyperparameters (see appendix for more details) for nezha-gen to generate texts. additionally, we use a random sampling strategy with top @xmath0. for setting k, we have analyzed 500 examples with different values of k, and found that adjusting k has a reasonable impact on the percentage of redundancy errors. except for the extreme case of @xmath1, the types of errors and the distribution of them do not change significantly. take commonsense errors as an example, which is the biggest difference from human-written texts. when k varies in a range of {5, 10, 20, 30, 50}, the percentage of commonsense errors is 18.6% ± 5.8%. redundancy errors account @xmath2.8%), but sharply drop to 37.4% as @xmath3, and the form of repetition changes from same-word repetition to a mixed repetition of "synonymous/same-word", suggesting that a simple repetition penalty may not be sufficient to deal with semantic redundancy. when @xmath4, 20, 30, 50}, the percentage of redundancy errors is very close to the result reported in figure 2 . when @xmath5, many generated sentences are completely incomprehensible. a larger k will also reduce the generation efficiency. therefore, we chose a sampling strategy of @xmath6, which is the trade-off between text quality and generation efficiency. ## conclusions in this paper, we have presented tgea, the first dataset with a variety of manual annotations on errors occurring texts generated by pretrained lan-guage models. for each erroneous text generated by a chinese gpt-2 model, our crowdsourced annotators detect erroneous text spans with their associated text spans and provide error types defined in a two-level hierarchical taxonomy as well as rationales behind detected errors. we elaborate the 5 annotation stages for building tgea with a strict annotation quality control protocol. we also report baseline results of the 5 benchmark tasks on tgea. the low results suggest that our dataset is a challenging testbed for future work on automatic detection of erroneous spans and types as well as producing error corrections and rationales for texts generated by plms. tgea is featured with wide error type coverage, rich semantic annotation and functional diversity, which can not only be used for deep diagnostic analysis on the text generation capability of pretrained language models, but also facilitate and promote the research of automatic and interpretable error correction for plm-generated texts.