Evaluating Large Language Models on Controlled Generation Tasks (2023)

Abstract: While recent studies have looked into the abilities of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc, there have been few studies looking into the controllability of large language models on generation tasks. We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models. We conclude that *large language models struggle at meeting fine-grained hard constraints*.

## introduction text generation models should generate texts that meet controllable constraints as humans wish @xcite . for example, one can avoid the blandness caused by repetitive patterns by controlling the syntax of generated sentences @xcite . in a customized dialogue system, one should be able to control the persona of the utterance @xcite . previous works either finetune generation models such as bart @xcite on specific tasks for better controllability (e.g., controlled paraphrase generation @xcite ) or design constrained decoding strategies (e.g., look-back decoding strategy by @xcite ) for controlled generation. large language models (llms) have recently shown great potential in various generation tasks. for example, @xcite shows that chatgpt with gpt-4 as an engine achieves commercial-level machine translation quality. @xcite find that annotators prefer summaries generated from chatgpt over state-of-the-art summarization models. however, task control benchmark evaluation numerical planning prefix & number of words & end word npb mse, success rate constrained content generation sentiment, topic, keyword amazon review commongen m2d2 off-the-shelf model, ppl story generation prefix rationale generation correct answer roc writing prompts repetition, diversity, coherence cos-e ecqa increased accuracy paraphrase generation semantic & syntax paranmt qqppos lexical overlapping, syntax match good poor figure 1 : we test large language models on five controlled generation tasks with various control factors using automatic evaluation methods. we show a spectrum of abilities of large language models on such tasks and conclude that large language models struggle at finegrained hard constraints such as numerical planning. the main contribution of this work is to conduct a comprehensive analysis of llm's controllability on five tasks and ten generation benchmarks, including controlled story generation, controlled free-form generation with sentiment and topics, controlled paraphrase generation, and controlled rationale generation as in figure 1 . we further design a new simple yet challenging benchmark named numerical planning benchmark (npb), where the task is to satisfy numerical constraints from four granularities (word-, syllable-, sentenceand paragraph-level) and under different content controls (e.g., prefix and ending). for evaluation, we use automatic metrics, which are imperfect yet convenient and reproducible. 1 after an in-depth examination, we categorize llm's controllability on a spectrum: from lagging behind and being on par with to surpassing smaller finetuned models. our findings indicate that large language models have difficulties adhering to specific hard constraints, such as numerical planning. we first introduce the numerical planning task and the associated evaluation as this is a new, intuitively simple, yet challenging task ( §2). for the rest, we rank them by the task difficulty indicated in figure 1 ## numerical planning can llms count from two to ten? task description. we introduce the numerical planning benchmark (npb) as an intuitive task that tests the basic numerical planning ability of llms. the high-level task descriptions can be found in evaluation metrics. we use success rate (sr) and mean squared error (mse) as automatic evaluation metrics. as our control is two-fold, we separately calculate the success rates of 1) generating the continuation with the correct counts and 2) generating the continuation with the proper ending. we also calculate the mse between our input numbers and output numbers. evaluate with llms. we evaluate chatgpt and alpaca-7b on our npb benchmark in zero-shot and few-shot settings. each request used to query the llms corresponds to a real case in the datasets llm-controlgen ## content-controlled generation task description. we consider three types of content constraints: topic, sentiment, and keyword. the detailed task definitions and dataset can be found in appendix b. evaluation metrics. we use the success rate as the evaluation metric to measure how well llms can follow the content constraints. specifically, we use gpt-3.5 @xcite based topic/sentiment classifiers with in-context learning using five examples per category to evaluate whether the generated texts belong to the specified topic or sentiment class. we consider an llm to succeed in one example if the predicted class of the generated text is identical to the input constraint. for a keyword-constrained generation, we use the keyword coverage metric that measures the percentage of input keywords included in generated texts. ## story generation task description. given the beginning text of a story, open-ended story generation aims to decode texts that are coherent with previous topics, and informative without undesired repetitions @xcite @xcite . despite the impressive success on generating fluent and accurate sentences for low-entropy tasks such as summarization or translation, large-scale language models (llms) still suffer from serious degeneration problems, such as undesired repetitions @xcite and unnatural topic drifts @xcite ), under openended settings. datasets. we evaluate different generation methods on two popular benchmark story datasets: rocstories and writing prompts. rocstories (roc) @xcite is a corpus comprising commonsense stories written by crowdsourced workers within 5 short sentences. given the first sentence as a prefix, generation methods are required to produce four continuing sentences. writing prompts (wp) is a challenging task for inspiring continuations with abstract, high-level story prompts submitted by online users and continuations by others on reddit @xcite . following prior literature @xcite , we utilize the first 32 tokens as the prefix and ask for continuation with 256 tokens. since we prompt different language models or decoding algorithms without extra fine-tuning, we directly sample 1,000 development and 1,000 testing instances from both roc and wp. baselines. we evaluate the pre-trained llm, gpt-2-xl @xcite , with both search (simctg @xcite and look-back (xu et al., 2023b)) and sampling decoding methods (nucleus sampling @xcite , typical decoding @xcite and η-sampling (hewitt et al., 2022)). ## rationale generation task description. free-form rationales are known to aid model interpretability by providing additional world knowledge or commonsense reasoning steps @xcite @xcite . @xcite show that rationales can improve large language models' ability to solve complex reasoning tasks. extractive rationales in question-answering tasks are based on the input passage to extract related information to answer the question. conversely, free-form rationales in the question-answering tasks are open- ## discussion: why and how we believe that our work makes a substantial contribution to the field of benchmarking llms' controllabiltiy, especially considering the prevalence of llms these days. that being said, we do have a few hypotheses to investigate why llms fail at numerical planning and how we could potentially increase their controllability. tokenization. on one hand, tokenization indeed makes the task of numerical planning more challenging than without, by separating the generative process (i.e., subword-level generation) and the numerical planning process (i.e., counting complete words). however, we posit that tokenizers not necessarily impact the ability of word planning, as it is a standard practice that a subword starting with a special token will indicate the start of a new word (e.g., " ġ" in bpe tokenizer, foot_4 which has been used by many llms such as gpt and roberta). nor are we aware of evidence that the subwords of a tokenizer roughly correspond to units of syllables. for example, @xcite shows that smaller models such as gpt-2-large fine-tuned on syllablerelated data can achieve a success rate of close to 90% on the same syllable-planning task. on the other hand, the best performance of chatgpt is 37%. decoding methods. the reported results are based on sampling with a temperature of 0.3. moreover, we have experiments showing that our conclusion is robust to the change of decoding mechanisms, where we try other decoding methods beyond sampling with @xmath0.3. specifically, we tried 1) greedy decoding, 2) beam search with beam size 8, and 3) sampling with temperature @xmath1.3, 0.7, 1.0}. for the prior two, most of the generated outputs are highly similar, plain, and lack diversity. as for sampling with @xmath2.3, 0.7, 1.0}, the success rate decreases as t increases. we think @xmath3.3 is a reasonable balance between diversity and quality. we believe that our results convey meaningful signals since each number n has been averaged over 100 different evaluation samples to reduce noise. however, none of these experiments show that llms can do better than fine-tuned gpt-2. in-context learning. we try to give more demonstration of npb in our prompts and we surprisingly found that this does not help once the input number n cannot be found in the examples. our results resonate with @xcite ; @xcite that llms do not truly understand task definitions via in-context learning. how to improve. we encourage future work to explore from two different directions: 1) chain/tree/graph-of-thought reasoning, and 2) bridging llms with non-autoregressive generation abilities (e.g., nado @xcite ). for the first one, one can try both simple chain/tree/graphof-thought prompting or even pretrained llms with chain-of-thought/scratchpad pairs, as it shows promises for mathematical reasoning @xcite . however, this will not fundamentally solve the planning issue. it is straightforward that autoregressively generating the next tokens will lead to the problem of models not "looking back" and therefore not adhering to the fine-grained control signals. therefore, we encourage researchers to also investigate multi-step planning and iterative revisions with llms, or, more fundamentally, challenge the autoregressive architecture of llms. ## conclusion we test the controllability of large language models on five tasks and ten benchmarks, including a numerical planning benchmark that is easy for humans while challenging for llms. from there, we draw a spectrum by comparing the performance between llms and smaller specialized models. llms are able to generate human-level rationales and conform with coarse control signals, such as sentiment, topic and keyword incorporation. however, they struggle at fine-grained hard constraints, such as numerical planning and paraphrase generations. we hope that our work can inspire downstream applications on when to adopt llms. for example, we find that llms are good at generating rationales, and these automatic rationales could be used to further boost llms' performance through chain-of-thought reasoning. certain sentiments. the aforementioned methods for topic constrained generation also apply to sentiment constrained generation.