Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints (2023)

Abstract: The limits of open-ended generative models are unclear, yet increasingly important. What causes them to succeed and what causes them to fail? In this paper, we take a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models. We present a generic methodology of analysis with two challenging prompt constraint types: structural and stylistic. These constraint types are categorized into a set of well-defined constraints that are analyzable by a single prompt. We then systematically create a diverse set of simple, natural, and useful prompts to robustly analyze each individual constraint. Using the GPT-3 text-davinci-002 model as a case study, we generate outputs from our collection of prompts and analyze the model’s generative failures. We also show the generalizability of our proposed method on other large models like BLOOM and OPT. Our results and our in-context mitigation strategies reveal open challenges for future research.

## introduction the recent success of large language models (llm) @xcite @xcite has transformed the field of natural language processing (nlp). in particular, prompting llms to generate open-ended text has shown promising performance. the existing and potential applications of open-ended text generation are farreaching, spanning domains such as qa @xcite , story generation @xcite , code generation @xcite , human-assisted creativity @xcite , open-ended dialogue @xcite , and the varied usages of chatgpt 1 . however, as llms continue to rise, there is a growing amount of concern over the unpredictability of nlp systems, and thus a need to better understand their capabilities and limitations. an extensive analysis of open-ended text generation is imperative to understand their capabilities, limitations, and areas for improvement. current analyses of open-ended text generation center around general text attributes, such as grammar, coherence, and toxicity. one way to categorize prompts is by their constraints. the prompt "create a short and funny joke about research" contains a variety of constraints. the output must be a joke (documenttype constraint), short (structural constraint), funny (stylistic constraint), and about research (subject constraint). the space of open-ended generative prompts can be partitioned by their constraints because all prompts are combinations of different types of constraints. in this paper, we systematically evaluate model performance on prompts that contain stylistic and structural constraints. a stylistic constraint bounds the style of the output, such as writing with a flowery style, and a structural constraint bounds the structure of the output, such as limiting the number of words in an output. we chose to analyze stylistic and structural constraints because they are omnipresent across prompts and notably challenging in literature @xcite . from formal emails to funny jokes, many generative applications have style. additionally, all generations have implicit or explicit structural constraints, such as length and proper formatting of an email or resume, and it is crucial for llms to understand them. we create simple, natural, and useful base prompts for each category, and vary them in a number of dimensions to ensure a fine-grained and robust analysis of each category. we use the public gpt-3 model as a case study to demonstrate the effectiveness of our proposed taxonomy foot_0 and comprehensively analyze the results quantitatively and qualitatively. we then test in-context mitigation strategies and provide directions for future research on the evaluation of open-ended generation. in summary, our contributions are as follows: analysis of large language models many existing benchmarks have been utilized to measure an llm's capabilities in natural language understanding and generation @xcite @xcite @xcite , where expected outputs are mostly deterministic and/or short. there is also much research analyzing general text attributes of open-ended text generations such as grammar, coherence, and toxicity. dhamala et al. (2021) uses automated metrics to test for gender bias, toxicity, and sentiment in a vast array of wikipedia-based prompts. dou et al. (2021) creates a framework that analyzes gpt-3 outputs for language errors, factual errors, or reader issues (such as usage of technical jargon). additionally, many studies use hand-crafted prompts to adversarially evaluate open-ended text generation models. chowdhery et al. (2022) uses the prompt "all x are " and calculates the average toxicity of continuations to evaluate palm's bias against group x. gehman et al. (2020) designs prompts that encourage toxic behavior from a model. lin et al. (2021) creates a dataset of handcurated prompts that elicit model hallucinations from gpt-3. in contrast, our goal is to investigate the open text generation capabilities of llms with regard to constraints in the prompt because we seek a more nuanced and bounded understanding of model performance. aspects like toxicity and grammatically are important across all outputs, but they don't provide insight into how correctly an llm responded to a prompt. controllable text generation controlling model outputs to fit a set of constraints is in the domain of controllable text generation. chan et al. (2020) uses a content adapter to control model outputs. krause et al. (2020) uses contrastive decoding to create generations with stylistic or topic constraints. keskar et al. ( 2019 ) finetunes an llm with inputs concatenated with an associated style token. however, creating these constraintcentric outputs requires a matching dataset of constrained text and an architectural shift. we evaluate controllable generation purely in-context and use comprehensive taxonomies instead of limiting evaluations to existing datasets. most similar to our paper, @xcite uses gpt-3 prompts to stylistically modify text and ask human raters to evaluate generation quality. in contrast, we provide a fine-grained analysis of model performance on generating styled texts. additionally, we focus on creating a set of simple, natural, and useful prompts for analysis. our goal is to understand the current capabilities and limitations of open-ended generative models. ## methodology the first step is to break down the constraint type into a taxonomy of individual constraints. these individual constraints must be analyzable by a single prompt with clear definitions of failure and success. we create our taxonomies by considering how users naturally put constraints in prompts. ## stylistic constraints stylistic constraints are present in all languages. these stylistic modifications often comprise of an adjective prior to a document type: "write a formal email to my boss; write a funny pickup line". stylistic constraints are notably challenging for llms in zero-shot settings @xcite . our stylistic constraints are grounded on existing work in the domain of reader's advisory (ra). ra takes a user-centric approach to recommending books based on their stylistic features. an ra taxonomy by function covers a diversity of stylistic text features that could be useful for both a writer and an audience. we use a comprehensive ra taxonomy found in pera and ng (2014). these features are writing style, tone, mood, characterization, pacing, plot, and genre. 4 each selected feature is used to stylistically modify text in unique and powerful dimensions. ## structural constraints structural constraints are omnipresent: "write an essay in fewer than 1000 words; limit your paper to 8 pages". structural constraints are notably challenging for llms @xcite . structure in the field of nlp is a broad term. we specifically analyze structural aspects of the text that are orthogonal to the actual content of the output. this includes length, spacing, and formatting, and excludes content-centric attributes such as syntax or semantics. our taxonomy is based on how a user could conceivably request a structural constraint within their prompt. we choose to analyze numerical, descriptive, and formatting structural constraints in this paper, but we note that this is not comprehensive of the entire structural space. ## conclusion we present a generic methodology to analyze a language model's ability to generate open-ended text under structural and stylistic constraints. our results show many failures that align with noted model challenges as well as new patterns of failure across structural and stylistic constraints. our sensitivity studies on model size show a rising trend rather than the emergence @xcite of stylistic and structural constraints. our mitigations demonstrate that adding additional in-context information consistently improves performance across both domains. future work could expand our work to look at other constraint types and more sophisticated mitigation strategies.