C hat GPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer (2023)

Abstract: Large-scale language models, like ChatGPT, have garnered significant media attention and stunned the public with their remarkable capacity for generating coherent text from short natural language prompts. In this paper, we aim to conduct a systematic inspection of ChatGPT’s performance in two controllable generation tasks, with respect to ChatGPT’s ability to adapt its output to different target audiences (expert vs. layman) and writing styles (formal vs. informal). Additionally, we evaluate the faithfulness of the generated text, and compare the model’s performance with human-authored texts. Our findings indicate that the stylistic variations produced by humans are considerably larger than those demonstrated by ChatGPT, and the generated texts diverge from human samples in several characteristics, such as the distribution of word types. Moreover, we observe that ChatGPT sometimes incorporates factual errors or hallucinations when adapting the text to suit a specific style.

## introduction generative pre-trained transformer (gpt; e.g., chatgpt) models, which produce results from given conditional input prompts, have exhibited exceptional performance on various natural language understanding (nlu) and generation (nlg) tasks @xcite @xcite @xcite . for instance, in nlu tasks, @xcite have proved that chatgpt is comparable to state-ofthe-art fine-tuning models in language reasoning. in nlg tasks, @xcite assessed four widely used benchmark datasets, such as qmsum, and confirmed chatgpt's comparability to traditional fine-tuning methods. @xcite further investigated effective strategies for machine translation using chatgpt and highlight its strong translation ability. additionally, chatgpt can even facilitate multi-modal tasks @xcite , as well as the application of data augmentation @xcite . although the studies mentioned above have demonstrated notable performance of chatgpt across different domains, there remains a dearth of qualitative and quantitative evaluation of the texts generated by chatgpt. such an evaluation is vital to uncover the behavioral differences, potential limitations, and challenges associated with chatgpt-generated texts, especially when compared with human-authored texts. controllable text generation seems to be a task in which chatgpt-like models could potentially excel. this task is driven by the desire to tailor text for a diverse array of target users (e.g., experts and laypersons) @xcite @xcite , and thereby enhancing the accessibility of textual information. in controllable text generation, one delineates a particular set of parameters or provides a prompt that defines the intended target style. this area has recently received growing interest from researchers in the field @xcite @xcite @xcite @xcite @xcite . the traditional natural language generation task @xcite , which focuses solely on adequately responding with respect to a given input, can be regarded as a special case of controllable natural language generation, wherein the control setting remains unconditioned. considering chatgpt as the most recent language generation capability, the assessment of its language generation proficiency, specifically in the realm of controllable language generation, remains largely uncharted. therefore, our study delves into two distinct applications of chatgpt, namely controllable summary generation and sentence style trans-1 fer. in the former, we examine chatgpt's ability to generate summaries that cater to two distinct readerships, namely experts and non-experts, for a given academic literature. concerning sentence style transfer, we investigate chatgpt's capability to generate both formal and informal sentences for the task of sentence formality. the objective of this study is to tackle the research question: in relation to the humanproduced text, to what extent does chatgptgenerated content demonstrate significant divergence from human behavior and the potential susceptibility to inaccuracies? our primary contributions are enumerated below: ## conclusion this paper presents a broad assessment of chat-gpt's proficiency in generating controllable text. we conducted quantitative and qualitative examinations at the document level (summarization task) and sentence level (text style transfer). the empirical findings show that chatgpt outperforms the previous state-of-the-art models on automatic metrics, but that there are substantial disparities between its generated texts and human-written texts. these disparities are reduced by providing a target example of the human writing style. furthermore, our investigations also confirm the previously reported problems of hallucinations and inaccuracies in text generated by chatgpt.