Systematic Task Exploration with LLM s: A Study in Citation Text Generation (2024)

Abstract: Large language models (LLMs) bring unprecedented flexibility in defining and executing complex, creative natural language generation (NLG) tasks. Yet, this flexibility brings new challenges, as it introduces new degrees of freedom in formulating the task inputs and instructions and in evaluating model performance. To facilitate the exploration of creative NLG tasks, we propose a three-component research framework that consists of systematic input manipulation, reference data, and output measurement. We use this framework to explore citation text generation – a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric and has not yet been tackled within the LLM paradigm. Our results highlight the importance of systematically investigating both task instruction and input configuration when prompting LLMs, and reveal non-trivial relationships between different evaluation metrics used for citation text generation. Additional human generation and human evaluation experiments provide new qualitative insights into the task to guide future research in citation text generation. We make our code and data publicly available.

## introduction thanks to their instruction-following abilities, large language models (llms) allow specifying and executing nlp tasks with unprecedented flexibility and speed, while reducing the need for task-specific architecture design, data annotation, and model training @xcite @xcite @xcite . this has led to a surge of new, complex, creative natural language generation (nlg) tasks like peer review generation @xcite or story and poetry generation @xcite , that push the boundary of what was deemed feasible for nlp systems just a few years ago. the flexibility comes at a cost, as it introduces new degrees of freedom into the analysis. llms generate output in response to a prompt, which consists of a natural-language task instruction supplemented by additional bits of information about an instance, which we term input components (figure 2 ). llm-powered creative nlg tasks often feature a complex input component space, and the task instruction wording can affect model behavior in non-intuitive ways. the output space is varied as well, as there might exist infinitely many acceptable generations. this overall variability brings the risk of creative nlg tasks being defined and evaluated ad hoc, hindering systematic comparison of nlp systems and leading to anecdotal accounts of llm capabilities. although optimizing model instructions to maximize performance of llms is an active research area (section 2.1), prompt engineering mostly tar-main paper abstract: this paper proposes a combined model for pos tagging <...> relevant paper abstract: in this paper, we propose a novel decoding algorithm for <...> intent: to compare the results of the proposed model with the results of the previous work. example: [ref#1] proposed a joint decoder for word segmentation, pos tagging and word-based constituent parsing, although they trained models for the three tasks separately. your aim is to generate an exactly single paragraph to be used in related work section in a main paper. you will be given main paper's abstract, a relevant paper's abstract and the intent of the paragraph. the paragraph should reflect the intent and you need to refer the relevant paper in the same paragraph by using citation mark @xcite . you can inspire from the given example. your output must strictly consist of the related work paragraph only, nothing else. gets the tasks where input and output spaces are well-defined (e.g., question answering). however, some creative nlg tasks need a step of exploration of what inputs are required and how the evaluation of outputs will be carried out before deeply exploring the best way to introduce the task to llms. our work addresses task variability in citation text generation -a widely studied scholarly nlg task aiming to increase efficiency of scientific work @xcite . citation text generation is a good example of creative nlg, as it features a complex input component space combined with multiple plausible outputs. prior work on citation text generation lacks consensus on the required inputs, explores only a limited number of measurements to characterize the outputs, and does not investigate the use of instructiontuned llms to tackle the task ( to address this gap, we design a framework to systematically explore the task of citation text generation with llms (figure 1 ). we systematically manipulate the input components and instructions communicated to the model via a prompt, and study the effects of these manipulations on the model output using a wide range of measurements, supplemented by a novel reference dataset for citation text generation based on the acl anthology, and featuring novel use of free-form citation intents to guide generation (section 3). our experiments with two state-of-the-art llms -llama 2-chat @xcite and gpt 3.5 turbo @xcite reveal that input components and task instructions both impact the generations, and their effects add up. free-form citation intents, as illustrated in figure 2 , show promise as an alternative to categorical intents used in prior citation text generation work. our results (section 5) imply that the relative performance of alternative task input configurations can be estimated on a small set of instructions, while the best absolute performance needs experimentation with a wide array of instruction wordings. through correlation analysis, we observe that the nlg metrics in our measurements are complementary, motivating the use of widespanning measurement sets for nlg tasks beyond citation text generation. our human studies (section 6) reveal both quantitative and qualitative insights about input components and task instructions from both generation and evaluation perspectives. in summary, this work contributes: • a framework for exploring the task of citation text generation with llms; • a new reference corpus of citation texts based on the acl anthology enriched with novel freeform citation intents; • experimental results on the impact of task inputs and instructions on citation text generation outputs, and an examination of the relationships between the measurements; • human evaluation and generation studies providing additional insights to shape future work in citation text generation and creative nlg. we stress that our work neither seeks nor claims state-of-the-art citation text generation, as the differences in pre-trained model capabilities would hinder a fair comparison and likely lead to confounding @xcite . instead, the objective of our work is to explore prompting as a tool for systematic task manipulation in the llm age. we believe our approach to be general and adaptable to other creative nlg tasks. ## task and method to recap, we aim to explore the impact of task input configuration and instructions on citation text generation outputs in the context of state-of-the-art study level abstract intent example model evaluation (abura'ed et al., 2020) sent tgt --pg rouge (xing et al., 2020) sent tgt --pg rouge, human @xcite sent tgt c -enc. + lstm rouge, human @xcite para tgt --fid rouge, human @xcite para tgt --hier. enc. rouge, human @xcite span tgt --led rouge, human @xcite sent src/tgt --gpt-2 rouge, bleu, human @xcite para src/tgt --pg rouge, human @xcite sent src/tgt c -t5 rouge @xcite sent src/tgt c -t5, bart rouge, scibertscore, human @xcite al., 2021) para src/tgt c -fid rouge, bleu, bleurt, meteor ours para src/tgt f ✓ llama 2-chat rouge, (sci)bertscore, bleurt gpt-3.5 surface, true, summac, human llms. we focus on paragraph-level generation for the related work section paragraphs, as this represents the dominant use case for citation text generation @xcite @xcite . the key components of our framework are prompt manipulation, reference data, and the measurement kit. ## experiments we experiment with two state-of-the-art llms, open llama 2-chat 13b @xcite and closed gpt 3.5 turbo @xcite (gpt-3.5-turbo-0613-16k). we dynamically construct our prompts according to the chosen input configuration. the outputs are analyzed using the measurement kit described above. to keep task complexity and computational costs at bay, we focus on the paragraphs discussing a single cited paper, resulting in 2,729 data instances. while using closed commercial llms is commonplace in nlp, this comes with reproducibility risks @xcite . to account for this, our llama 2-chat experiments were run on-site, taking approximately ∼30 hours on a single nvidia a100 gpu with 80gb memory, and are strictly reproducible. we used flant5-xxl foot_2 @xcite to generate free-form intents for each citation paragraph in our dataset, and the fine-tuned multicite model foot_3 @xcite to assign categorical intents. further details are given in appendix c. we provide example model generations in appendix e. ## results our framework allows us to answer a range of questions about citation text generation in the context of modern llms. here we focus on a subset of these questions and provide exhaustive experimental results in the appendix c. we use the following notation to discuss experimental configurations: #(+a)(+ic/if)(+e), where # is the instruction template identifier ranging from 1 to 6 (appendix b), +a denotes source and target paper abstracts, +ic and +if denote the categorical and free-form intents, and +e denotes an example citation sentence that cites the given paper. the instructions are adjusted to reflect the input components present in a given configuration. the example input in figure 2 corresponds to the configuration 1+a+if+e. we concatenated both cited and citing abstracts and used it as baseline. rq1: what is the impact of the input configuration on generated texts? figure 3 presents our main results across different configurations. for all conventional metrics (rogue, (sci)bertscore, bleurt), all configurations far outperform the baseline. for true, while gpt 3.5 scores higher than baseline except for one configuration, llama-2 performs close to the baseline. on the other hand, baseline summac scores are better than both model outputs. we also observe that providing the model with only abstracts (+a) yields the lowest degree of correspondence between the generated text and the reference in the vast majority of the configurations. this suggests that additional free-form intents and example sentences have a positive influence on performance in terms of both conventional and nlibased measurements. we emphasize that this effect is consistent across different types of models and prompts. we also observe that providing models with free-form intents (+if) increases the correspondence between generated and reference citation texts more than example sentences (+e). jointly providing free-form intent and example (+if+e) shows a combined effect and yields the best correspondence in 90.28% of 72 measurements (two models × six prompts × six metrics). we additionally investigated whether this effect is only due to the input length and observed that even shorter +a+if+e instances are still significantly (@xmath0.05) better than longer +a instances. details of the experiments controlled for prompt length can be found in appendix d. overall, we note that the ranking of configurations remains mostly consistent across the task instructions and measurements. this suggests that the relative performance of different input configurations might be estimated based on a small number of instruction variations. rq3: how does the intent type affect the outputs? figure 3 allows us to compare the performance of free-form and categorical intents across different experimental settings. we observe that categorical intents are insufficient to generate paragraphs that are close to the reference paragraphs. categorical intent combinations with abstracts (a+ic) perform not better than only abstracts (a) in almost all cases. furthermore, categorical intent combinations with example sentences (a+ic+e) are outperformed by using free-form intents (a+if), which holds for all measures and instructions, and for both investigated llms. this preliminary evidence suggests that while free-form intents are helpful, categorical intents might not be effective input components for citation text generation with llms. rq4: what are the relationships between the measurements? we observe that conventional metrics negatively correlate with the generated output length, while nli metrics show little to no correlation (figure 4 ). this indicates that longer outputs per se do not result in having more similar meaning to gold reference and obtaining higher scores -in line with the decreasing trend of rouge score afconfiguration rou ber sci ble tru sum [h] 6+a 14.16 85.69 61.20 37.36 10.00 21.33 [h] 6+a+if+e 16.25 85.88 61.81 38.56 13.33 22.00 ## human evaluation human-generated citation texts. to get further insights into the impact of input components on citation text generation, we conducted a small-scale human generation study. we sampled 30 instances from reference data used in our main experiments, for which three annotators with nlp backgrounds composed related work paragraphs. for each instance, annotators were first given the abstract information along with instruction (6+a) and wrote the citation paragraphs. then, they repeated the same process with the addition of free-form intents and example sentences, corresponding to (6+a+if+e). the annotators had no access to gold reference paragraphs. the human-written texts were then measured using the metrics in our kit and compared against the reference. evaluation. for the same set of instances, the respective llama 2-chat and gpt 3.5 outputs were manually evaluated by the annotators in terms of their correspondence to the gold reference. we used the pyramid method @xcite , in which the generated output is compared to the gold reference in terms of its compliance with a set of basic facts derived from the reference. we extracted atomic facts from the reference paragraphs using gpt-3.5 (details in appendix f). the facts were manually quality-controlled and curated prior to evaluation. then, for each citation paragraph instance, annotators checked whether the generated paragraphs mention the extracted atomic facts. the model and input configuration for the generated outputs were not known to the annotators, and the order of presentation of outputs was randomized for each question. we note that human evaluation doesn't consistently favor any of the two llms, with the best results obtained by gpt 3.5 turbo. yet, we note that this might be due to the small scale of the human evaluation study, and leave further exploration of this discrepancy to the future. further details of the both human generation and evaluation can be found in appendix f. we additionally investigated the correlation between human evaluation results and other metrics. qualitative observations. our evaluations yielded few informal insights which we deem useful for follow-up research. llm generations were typically more verbose (see as this is not reflected in the metric performance scores, we hypothesize that pragmatic mismatch might not be captured by the automatic evaluation metrics. we found that the success of generations depended on the content of the gold reference: while high-level discussion of related work can be generated from the abstracts, going into specifics of a paper requires information not available in the input. uninformative abstracts were also hard to generate from, both for humans (who wrote short and uninformative citation texts in response) and for llms (that were forced to hallucinate text). since the setting of our human study is insufficient to investigate these observations empirically, we leave this exploration for future research. ## conclusion the last generation of llms has enabled a wide range of novel, creative nlg tasks characterized by flexible input component space and a wide range of plausible outputs. this flexibility brings about new challenges as it introduces additional degrees of freedom into the experimental setup, but it also allows nlp researchers to systematically study the relationships between the input components, the instructions, and the generated outputs. in this work, we have addressed the variability of creative nlg tasks by systematically exploring the task of citation text generation. we proposed a framework for systematically comparing task configurations and used it to study the impact of task input and instruction on the citation text generation performance, by both llms and humans, using a widespanning range of measurements to characterize the llm outputs. our insights contribute to a better understanding of the role of input configurations, instructions, and output measurements in llm-based language processing, and our framework facilitates the study of citation text generation in the age of llms. our human evaluations provide additional insights to guide future work. in an environment where anecdotal evidence of llms' impressive capabilities is overabundant, our work contributes to the best practice for the systematic study of llm-based approaches to complex, creative text-generation tasks.