Controllable Citation Sentence Generation with Language Models (2024)

Abstract: Citation generation aims to generate a citation sentence that refers to a chosen paper in the context of a manuscript. However, a rigid citation generation process is at odds with an author’s desire to control specific attributes, such as 1) the citation intent, e.g., either introducing background information or comparing results, and 2) keywords that should appear in the citation text. To provide these degrees of controllability during citation generation, we propose to integrate the manuscript context, the context of the referenced paper, and the desired control attributes into a structured template and use it to fine-tune a language model (LM) via next-token prediction. We then utilize Proximal Policy Optimization to directly optimize the LM in favor of a high score of our proposed controllability metric. The proposed workflow harmoniously combines citation attribute suggestion and conditional citation generation into one LM, allowing for better user control.

## introduction a common practice in scientific writing is to cite and discuss relevant papers that support an argument, provide background information, or compare results @xcite . recent studies aim to facilitate this citation process by using neural networks to generate a citation sentence based on the context of the manuscript and the paper to be cited. these efforts focused primarily on developing a sequence-to-sequence pipeline that works in a fully automated, uncontrolled manner, leaving little room for users to control the generation process. we believe control is desirable because authors often have a clear motivation before writing a citation sentence. for example, they may have a specific intent to cite, such as comparing results or presenting background information, or they may have keywords in mind to appear in the citation sentence. when the generated citation does not match an author's motivation, the author may wish to change the generation by specifying certain attributes, such as citation intent or keywords. this study aims to develop a citation generation model that is controllable, such that users can alter the citation intent or topic by explicitly providing the citation intent and keywords. our proposed method involves the following two phases: supervised fine-tuning. we design a structured prompt template that systematically and consecutively incorporates contextual information, citation attributes (intent and keywords), and the citation sentence into a sequence of tokens, and fine-tune an lm via next-token prediction. through supervised fine-tuning, the lm learns to generate citation sentences not only based on the manuscript/cited paper's context but also conditioned on the citation attributes, thus allowing flexible control of generation by altering the citation attributes @xcite . controllability enhancement via reinforcement learning. we propose measuring the controllability of a citation generation system from multiple aspects with the following metrics: i. intent alignment score (ias), which measures whether the intent of the generated citation sentence matches the given control intent; ii. keyword recall (kr), which measures the recall of the control keywords in the generated citation; iii. fluency score (fs); and iv. rouge-f1 @xcite score of the generated sentence compared with the groud-truth citation. these controllability evaluation metrics provide a further opportunity to guide the training of our system by using them to estimate a reward function for proximal policy optimization (ppo) @xcite . this allows us to explore the effect of using feedback from our chosen metrics to improve our system's controllability. our contributions are summarized as follows: previous work in citation generation, including xing et al. (2020); ge et al. (2021); wang et al. (2022), approached the task as a sequence-tosequence translation problem, utilizing recurrent networks and knowledge graph enhancements. nikiforovskaya et al. (2020) proposed a bertbased extractive summarizer (liu, 2019) that produces a paper review by extracting one sentence from each of the related papers. chen and zhuge (2019) proposed automatically generating a related work section by extracting information on how papers in the reference list have been cited in previous articles. xing et al. (2020) developed an rnn-based pointer generator network that can copy words from the manuscript and the abstract of the cited paper based on cross-attention. ge et al. (2021) further extended this work by enhancing citation generation using information from the citation graph. despite their advances, these approaches do not fully address the complexities of citation sentence generation. our research highlights the importance of user-specified attributes and illustrates the limitations of large language models in reliably inferring key attributes such as topic keywords. we propose that this semantic gap necessitates a shift towards models that take explicit user control into account in citation generation. recent research jung et al. (2022); wu et al. (2021); yu et al. ( 2022 ) has also explored controlled citation generation, though the focus is often limited to controlling a single attribute like citation intent. our work extends this concept to a broader range of attributes, introducing methods to suggest potential attributes and balance the automation and controllability of citation generation. we further differentiate our research by conducting extensive experiments with cutting-edge language models and investigating the augmentation of controllability via reinforcement learning, contributing to a more comprehensive understanding of controlled citation generation. in addition, we go beyond prompt-based approaches @xcite by investigating enhancing the controllability of our citation generation system with reinforcement learning. ## method we first describe our supervised method for finetuning an lm, and then we describe how we measure the control of the generated sentence by the attributes. finally, we introduce strategies for enhancing controllability through reinforcement learning. our method is illustrated in figure 1 . ## dataset from a subset of arxiv computer science papers, we extracted triplets consisting of the citing paper (treated as the manuscript), the citation sentence within it, and the corresponding cited paper. the necessary components for the input context, including the local and global contexts of the manuscript, as well as the global context of the cited paper, were then extracted from these triplets. regarding the citation attributes, we used the scibert-based intent scorer, as outlined in section 3.2, to predict the most probable citation intent for each citation sentence. to obtain the keywords attribute, we extracted noun phrases from each citation sentence and ranked them based on the cosine similarity between the keyword's sentence-bert (version "all-mpnet-base-v2", apache license 2.0) @xcite embeddings and the embeddings of both the manuscript and the cited paper. this approach allowed us to retrieve up to two keywords per citation sentence. to create the training, validation, and test sets, we used a chronological split. all the citing papers used in the training set were published before march 1st, 2023, while those in the validation and test sets were published after this date. this strategy prevents any unfair advantage for the tested language models (section 5) by ensuring they have not previously encountered the same papers and their associated citation sentences in the validation and test sets during pretraining. the statistics of our dataset are shown in ## experiment we experimented with the encoder-decoder model bart @xcite and decoder-only models including gpt-neo @xcite , galactica @xcite , and llama @xcite , which varied in size from 125m to 7b parameters. all models were pretrained with corpora before march 1st, 2023. during supervised fine-tuning, the smaller models (125m to 1.3b parameters) were fine-tuned in float16 preci-sion with a learning rate of 1e-5. for larger models like galactica-6.7b and llama-7b, we utilized int4 low-precision quantization @xcite and low-rank adaptation (lora) @xcite to reduce the gpu memory footprint. we set the lora parameters with the rank @xmath0. for these larger models, we used a higher learning rate of 1e-4, in line with the guidance from @xcite . all models were optimized with the adamw optimizer, using default betas (0.9, 0.999), weight decay 0.05, and cosine learning rate decay. models were fine-tuned for 5k steps with a batch size of 128 with a maximum token length of 1024. we selected the best model checkpoint based on the validation set loss every 1k steps. to enhance controllability, we subjected galactica and llama to additional fine-tuning using proximal policy optimization (ppo), as described in section 3.3. during this process, the language models generated batches of citation sentences (termed 'episodes') for subsequent mini-batch-wise backpropagation. we set the learning rate to 1.41e-5 for all ppo fine-tuning. the ppo-finetuned models are named galactica-125m-ppo, galactica-6.7b-ppo, and llama-7b-ppo, respectively ( in addition to fine-tuning models, we leveraged gpt-3.5-turbo-0301 foot_1 , the backbone of chatgpt, through prompt engineering. the aim was to compare our fine-tuned models' performance against a large language model with carefully crafted prompts. we designed specific prompts (detailed in appendix c) for the chatgpt api to generate citation sentences. we conducted this in controlled (providing context and citation attributes) and uncontrolled (providing only context) modes. ## results and discussion we aimed to investigate several key questions: 1) whether and to what extent controllability offers advantages in citation generation, 2) whether ppo helps to improve controllability, and 3) how the model size and the nature of pretraining tasks influence overall performance. in addition, we conducted a human evaluation to compare the controllability of our fine-tuned citation generator against gpt-3.5. this evaluation provided insights beyond ## conclusion in this study, we introduced a controllable citation generation framework that leverages language models, highlighting the importance of user-specified attributes in the generation process. we emphasized the necessity for attribute control, underlining the complexities of citation generation, and explored the potential of enhancing controllability through proximal policy optimization (ppo). our experiments affirmed that large language models pretrained on scientific corpora are essential for citation generation, with the fine-tuned model showing advantages over gpt-3.5-turbo in both automatic metrics and human evaluations. intent category (# samples) background method result average (1,014) (613) (260) (macro) jurgens et al. (2018) 84.7 74.7 78.2 79.2 cohan et al. (2019) 87.8 84.9 79.5 84.0 scibert+scaffolds (ours) 89.1 87.1 84.0 86.7