Uniform Complexity for Text Generation (2023)

Abstract: Large language models (LLMs) have shown promising results in a wide array of generative NLP tasks, such as summarization and machine translation. In the context of narrative generation, however, existing models still do not capture factors that contribute to producing consistent text. For instance, it is logical that a piece of text or a story should be uniformly readable throughout and that this form of complexity should be controllable. As such, if the complexity of an input text prompt is rated first-grade reading level in the Flesch Reading Ease test, then the generated text continuing the plot should also be within this range of complexity. With this in mind, we introduce Uniform Complexity for Text Generation (UCTG), a new benchmark test which raises the challenge of making generative models observe uniform linguistic properties with respect to prompts. We experiment with over 150+ linguistically and cognitively motivated features for evaluating text complexity in humans and generative models. From our results, we find that models such as GPT-2 struggle to preserve the complexity of input prompts used in its generations, even if finetuned with professionally written texts.

## introduction consistency is key in all aspects of the storywriting process. a story has to have a consistent plot or story arc, observe a consistent use of factual information if needed, make use of consistent personas for characters, and follow a consistent writing style in order to be comprehensible @xcite @xcite . while topic consistency is often used as a general metric for evaluating the quality of texts produced by narrative generation systems @xcite , an important aspect that has received less attention from the research community is the figure 1: an illustrated ideal example of a generative model (e.g. gpt-2) producing uniformly complex text where the flesch ease score of the generated text falls within the same readability range as the prompt which is 60-70 or 8th-9th grade reading level. when crafting a story, it is very reasonable for a writer to ensure that all sentences are readable by a specific target audience (i.e., first-grade learners). as such, one must take note of common words within the general vocabulary of readers belonging to a specific grade to avoid possible frustration in reading that will hinder effective comprehension @xcite . thus, a text's complexity (and so the readability) depends largely on the writer's capability to ensure that the complexity of words and sentences is consistent @xcite @xcite . in the text generation process, since the output of generative models is controlled by the prompt, it is essential that the complexity of the output to be consistent with the input prompt. figure 1 shows an ideal scenario where a prompt given by a human has the same level of reading difficulty (66.9) as the generated text by a gpt-2 model (61.9) based on the flesch ease formula @xcite . the interpretation of flesch ease scores obtained means both texts can be read easily by grade 8 and 9 learners. generally, the complexity of a narrative or any piece of text at hand can be measured by a multitude of content-based and linguistic factors. some measures that have been explored through the years include syntactic complexity as equivalence of high-proficiency writing through the presence of qualified syntactic phrases or chunks such as words from a noun or verb phrase @xcite @xcite , discourse complexity by aggregating the presence of entities of a text such as mentions people, organizations, and locations which can lead to increased working memory burden @xcite , and vocabulary complexity by matching words from the text associated with a specific age-based difficulty level from developmental studies @xcite to name a few. these factors, when extracted from texts for evaluating complexity, usually follow a general linear property where the complexity of a text also increases as the value of a measured feature is increased @xcite . in this study, we aim to answer the following research question: can large language models (llms) generate responses that observe uniform complexity with the prompt? to answer this, we introduce the novel task of uniform complexity for text generation (uctg) with the aim of investigating responses obtained from any arbitrary generative language models used in story generation with gpt-2 models as test case, we explore how the outputs generated by llms relate to prompts and continuations given by humans, in relation to these linguistic features. specifically, we make the following major contributions: the proposed evaluation task can be generally adapted to any nlg framework as long as texts are both the input and output. in the case of a narrative generation setting, given a series of prompts where each one is made up of a few sentences @xmath0, p 2 , . . . , p n } and their continuations @xmath1, c 2 , . . . , c n } by a model m , a linguistic feature function f processes instances from both the prompt and continuation sets to get their corresponding measures f (p n ) ∈ r and f (c n ) ∈ r. all calculations performed by f are normalized foot_1 with either the number of word counts or a similar variable to avoid higher values from longer generated texts. a statistical test is then applied using the two groups to measure the significant differences in each linguistic complexity feature variable. ## generation setup we describe the training recipes and resources used for exploring uctg in human and gpt-2 models. ## testing for (in)consistencies from the prompt-continuation pairs for the first experiment, we test for possible significant differences in the complexity values of the prompts and generations of the three variations of gpt-2 models described previously plus generations from humans. we use the welch t-test @xcite with bonferroni correction resulting in a new alpha level of α = 0.0125 (0.05/4). we extracted 160 linguistic features using the lingfeat tool by @xcite for the variables of interest. the tool covers the extraction of surface, lexical, phrasal, tree structure, type token, psycholinguistics, and formula-based readability features as further detailed in the subsections below. we only used the normalized or average-based linguistic complexity features (ex. count of noun pos / total count of words) instead of total raw counts to avoid bias or reaching extremely high values for longer sentences. as a reference for labels of models used in the succeeding tables, we use gpt2 for the base model with no fine-tuning, gpt2-df for the model fine-tuned for general narrative generation, and gpt2-uf for the model fine-tuned with uniformly complex dataset. for ease of readability and display purposes, we report the p-values from the result of the welch test for the tables per linguistic group in the succeeding sections. we also include an extensive set of barplot visualizations in appendix a arranged alphabetically for ease of referencing. ## similarities between human and model generations aside from prompt-wise comparison, we also look at which linguistic complexity features from the gpt-2 models have some form of correlation with human continuations. this experiment aims to investigate which neural story generation models are potentially closer in producing more humanlike text with respect to linguistic characteristics. for this, we use pearson correlation to do the continuation-wise analysis and extract the top correlated features described in the evaluation of natural language generation (nlg) systems is an important process toward ensuring their usability, accessibility, and accuracy in practice. this research area has become one of the most active in the nlp community in recent years due to the rise of more complex and powerful generative models @xcite . synthesizing the results shown from sections 4 and 5 in this study reporting the current inconsistencies of finetuned gpt-2 models in generating uniformly complex texts, one may reason out the potential of uctg as a benchmark task for existing nlg systems to gather attention and encourage work from the research community. for this proposition, we highlight three major points below as insights that may be considered for building more uniformly complex text generation models. ## conclusion in this study, we introduce uniform complexity for text generation (uctg), a new benchmark task to encourage the production of uniformly complex generations with respect to prompts. to further ground the existence of the problem, we perform the most extensive investigation of linguistic properties (160 features) analyzed from generations of gpt-2 models for the narrative generation task. we find clear inconsistencies with the models, whether finetuned or not, in terms of preserving the text complexity of generations with respect to the prompts used. however, we do note some similarities in specific linguistic variables related to word familiarity and word use between humans and gpt-2 models in how they produce continuations to a prompt. recognizing the potential and benefits of uctg to increase the usability and accessibility of nlg systems, we discuss three major recommendations covering the importance of user background, language dependency, and flexibility of implementation if uctg is to be implemented as an automatic evaluation metric or benchmark task for any arbitrary generative models from here onwards.