Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues Using BERT (2021)

Abstract: This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. While this task was previously rendered through expensive and time-consuming human labor, we present this novel task of automatic naturalness evaluation of generated language. By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT. In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge.

## introduction with the increasing popularity of virtual assistants such as alexa or siri, users have a higher demand for conducting natural conversations. they would like to chat with these assistants more naturallymaybe even like talking to a real human being. one of the key questions arising from this, though, is how to measure the naturalness of the generated language. in the past, native speakers judged the quality of the generated language by answering questions like "is this utterance natural?" or "could it have been produced by a native speaker?" to rate the naturalness @xcite . however, this approach heavily depends on manual effort and is impractical for broader use. on the other hand, the widely used automatic metrics for evaluating language generation, like bleu @xcite and meteor @xcite , rely on word overlap mechanism and compare the generated sentence to one or more human-created reference sentences @xcite , but cannot directly reflect naturalness information. likert scale ratings are generally used in human evaluation. the figure 1 shows an example of sentence: x is a moderately priced restaurant in x. 6-point likert scale rating on naturalness of a sentence. in general, the human evaluation items (naturalness, coherence, etc.) are hard explainable objectively and native speakers purely rely on their underlying criteria to judge the performance of generated language without any specified rules. to alleviate the manual effort previously needed, the goal of this paper is to render the task of assessing the naturalness of generated language as an automatic reference-based classification problem having each likert scale rating representing one class. to our knowledge, we are the first to propose this task and present an approach evaluating naturalness of generation through fine-tuning a pre-trained language model (lm). the contribution of this paper is two-fold: a pretrained bert model is fine-tuned to estimate the naturalness of a generated sentence based on a reference sentence. three baselines are proposed for robust performance comparison: a support vector machine (svm) baseline using bag-of-words (bow) vectors for input representation, a bi-directional long short term memory (lstm) neural network, and the pre-trained bleurt model @xcite . second, based on the positive correlation between naturalness and other annotated information: quality and informativeness of the generated sentence, the proposed method is extended to leverage this additional information through transfer learning. and the learning speed and estimation performance of naturalness evaluation model is significantly increased. the remainder of this paper is structured as fol-lows: section 2 introduces the related works. in section 3, our proposed approach for bert-based naturalness estimation is presented. in section 4, the experiment setups are covered, which include fine-tuning bert, comparison against the baselines, and transfer learning. section 5 describes the experiments results. the last section 6 draws conclusions and outlines future work. ## related work with the development of natural language generation (nlg) applications and their benchmark datasets, evaluation of nlg systems has become increasingly important. generally, multiple automatic metrics are used in parallel to evaluate the performance of language generation, such as bleu @xcite , meteor @xcite or rouge @xcite . however, @xcite demonstrated that the existing automatic metrics have poor instance-level correlation with mean human judgment and that they assign bad scores to many good quality responses. since automatic metrics still fall short of replicating human decisions @xcite , many nlg papers include some form of human evaluation. for example, @xcite report that 20 out of 26 generation papers published at acl2018 presented human evaluation results for showing their robust performance comparison. @xcite and @xcite also highlighted that human evaluation is commonly viewed as the best reliable way to evaluate nlg systems, but come with many issues, such as costly and time consuming and human judgement bias. and more importantly, the evaluation results from human efforts are not always repeatable @xcite . @xcite previously attempted to predict quality ratings of generated language by using recurrent neural network (rnn) with the help of the meaning representations (mrs) and showed promising performance. however, our work is more focused on naturalness evaluation and based on the gold reference. in order to relieve the human labor, we propose a reference-based method for naturalness evaluation by utilizing neural network to learn the complicated linguistic relationship from sentences. and this work can be easily extended to other human evaluation criteria (coherence, quality, etc..), if the corresponding human evaluation data is available. in recent years, with huge success of pretrained lms @xcite , many machine learned metrics for evaluating generation are proposed. especially the pretrained bert (bidirectional encoder representations from transformers) @xcite shows its superiority in this field. @xcite presented automatic machine translation evaluation by using bert and achieved the best performance in segment-level metrics tasks on the wmt17 dataset for all to-english language pairs. zhang et al. (2019) 3 naturalness evaluation using bert the task of estimation the naturalness of a generated sentence is framed as a classification task. two sentences are used as input: the candidate sentence to be scored and a reference sentence. the naturalness score is derived through fine-tuning a pre-trained bert @xcite model. the main architecture of bert uses the encoder of a transformer @xcite , which is an advanced encoder-decoder architecture leveraging the attention mechanism. considering that the input may be sentence pairs in several tasks, technical innovation next sentence prediction (nsp) helps bert to learn the relationship between sentence pairs by receiving pairs of sentences as input and separating them with [sep] token to learn predicting if the second sentence is the subsequent sentence in the original document. to do so, a @xcite token is added at the beginning for every input to learn the meaning of the entire input. exactly because of these specific characters, eleven nlp tasks figure 2 shows the fine-tuning structure for naturalness estimation with a candidate sentence (sys ref) and a reference sentence (orig ref) as input. example for a sys ref and orig ref is shown in ## experimental setup in this section, the experiment procedure including the pre-processing of the used data is introduced. first, bert is fine-tuned for naturalness estimation and then compared it with the baselines. furthermore, external knowledge is added demonstrating the impact on naturalness estimation through transfer learning. ## results and discussion the results of our proposed approach (bert) for estimating the naturalness of a generated sentence given an additional reference are depicted in the results in even though the data set is imbalanced as shown in given the imbalanced data set, the macro f1 score, recall and precision are also computed to show the robustness of our proposed approach. moreover, through the transfer of quality (or informativeness) knowledge to naturalness training, the performance of naturalness estimation is further improved and training speed has also been greatly promoted. figure 3 shows that the naturalness training based on transfer learning is faster and tends to be stable after only 5 epochs. the ## conclusion and future work in this paper, we proposed a novel task of automatically estimating the naturalness for task-oriented generation based on a human reference. we proposed a robust estimation approach by fine-tuning the pre-trained bert model which outperforms an svm classifier, bi-lstm, fine-tuned bleurt and majority class accuracy. taking advantage of the positive correlation of naturalness on quality (or informativeness), we successfully improved the naturalness training speed and estimation performance through transfer learning. this work sheds light on research towards naturalness evaluation by neural network learning. the final goal of our work is to relieve the human labors from naturalness evaluation task and realize the automatic naturalness evaluation for dialogue generation. hence, we will firstly collect more human evaluation data for future work. because the human evaluation data, which is already shared and public on the internet, is very limited. with more collected human evaluation data, we are also interested in the performance of our proposed method on other