Time-aware Prompting for Text Generation (2022)

Abstract: In this paper, we study the effects of incorporating timestamps, such as document creation dates, into generation systems. Two types of time-aware prompts are investigated: (1) textual prompts that encode document timestamps in natural language sentences; and (2) linear prompts that convert timestamps into continuous vectors. To explore extrapolation to future data points, we further introduce a new data-to-text generation dataset, TempWikiBio, containing more than 4 millions of chronologically ordered revisions of biographical articles from English Wikipedia, each paired with structured personal profiles.Through data-to-text generation on TempWikiBio, text-to-text generation on the content transfer dataset, and summarization on XSum,we show that linear prompts on encoder and textual prompts improve the generation quality on all datasets.Despite having less performance drop when testing on data drawn from a later time, linear prompts focus more on non-temporal information and are less sensitive to the given timestamps, according to human evaluations and sensitivity analyses.Meanwhile, textual prompts establish the association between the given timestamps and the output dates, yielding more factual temporal information in the output.

## introduction temporal information, such as publication and modification dates of documents, is an inherent attribute of documents. both document writers and readers are aware of this information when organizing and consuming document content. for example, an event reported by a news article is likely to happen right on the publication date. however, stateof-the-art generation models are fine-tuned from large pre-trained models without incorporating temporal information @xcite , creating a gap between document processing by humans and automatic models. though previous work has split datasets according to temporal information and shown deteriorated performance of large pre-trained models as knowledge becomes outdated on future data @xcite , it is unclear how informing models of temporal information affects generation tasks. therefore, this work aims to study the effects of presenting temporal information to generation models. concretely, to include timestamps in model inputs, we consider prepending two types of timeaware prompts to the encoder or the decoder. first, textual prompts encode timestamps within natural language descriptions, as commonly used by the recent prompt engineering work @xcite . we further explore linear prompts that map timestamps to continuous vectors via linear projections. concretely, we fine-tune bart (lewis et al., 2020) with time-aware prompts and conduct experiments on two text-to-text generation tasks: (1) content transfer @xcite that generates the continuation of a passage using information from a given document, and (2) summarizing news articles with xsum @xcite . to study time-aware prompts' capability of extrapolating to future dates, we introduce tempwikibio, a datato-text dataset containing timestamped revisions of biographical articles from english wikipedia, each paired with an infobox as input. the revisions record changes of personal profiles from 2004 to 2021. 1 for all experimented datasets, dated events are critical. we first evaluate model outputs with automatic metrics to examine the effects of temporal information on output informativeness. human judges are then asked to additionally rate the factuality of model outputs and determine if the improvement or degradation is due to date changes we highlight relevant temporal information in the input and corresponding correct (incorrect) information in the model outputs. linear prompts could encourage selecting important dates on tempwikibio, but the temporal information encoded in the linear prompts can not be captured by the model, leading to incorrect dates when resolving with the provided dates is required on xsum; while the model with textual prompts is sensitive to the provided dates and generates correct date, it lacks world knowledge (e.g., seasonal earning is only reported after the season) to handle the last case after perturbing the original publication date. in the outputs. finally, we perform a sensitivity analysis by making perturbations to the original dates (e.g., setting the dates to one year before) and providing models with the perturbed dates, and then inspect the changes of outputs. we find that: ## related work temporal generalization in nlp. early work on temporal generalization focuses on detecting the shifts of n-gram frequencies over time @xcite and detecting word meaning changes @xcite . besides linguistic shifts, model degradation on downstream tasks has been reported when tested on samples at a different time from the training data @xcite @xcite . in this work, we study the temporal generalization of our time-aware prompts, since they are constructed with temporal information. prompt engineering. prompts have been a common tool for controllable generation @xcite @xcite . instructions are also constructed as prompts to allow large models to perform new tasks that are unseen in training @xcite . more recently, prompts, either hand-crafted (schick and schütze, 2021; @xcite or learned (li and liang, 2021), are found to benefit model learning and improve few-shot performance on downstream tasks. our textual timeaware prompts extend the year-level prompts in @xcite with months and days to incorporate fine-grained temporal information, and we further explore representing timestamps with linear prompts which have been mainly used for lengthcontrolled generation @xcite . encoder decoder … year month day … (2018 -1990) / 50 2018-05-23 0.81 5 / 12 23 / 31 0.32 -0.11 -0.24 0.19 -0.47 0.21 0.73 -0.27 learnable weights normalization 0.45 0.18 -0.06 -0.10 0.08 -0.19 0.16 0.54 -0.20 prompt vectors figure 2: a linear prompt treats year/month/day as separate scalars and projects them into continuous prompt vectors to be used on encoder or decoder. the vector's scale reflects their temporal orderings. note that the dimension of prompt vectors is the same as the embedding dimension in the actual model. ## time-aware prompts we study two types of prompts that are prepended to the encoder/decoder of a seq2seq model, to inform the model of temporal information. ## tempwikibio data collection to study how well time-aware prompts can extrapolate to future data, we collect tempwik-ibio, which has 4,277,450 revisions of infoboxparagraph pairs from 2004 to 2021 for 695,929 wikipedia biography articles, extending wik-ibio @xcite , which only includes the latest revision per article by 2015. importantly, the profile (e.g., titles, awards, etc) of a person shown in the infobox changes over time (figure 3 ). concretely, for each biography, we pick its latest revision every x days since the first revision, where x is sampled uniformly from @xcite , to diversify the timestamps included in the data. we then extract the infobox and the lead paragraph per revision. as illustrated in figure 3 , two test sets are created: test-same time contains articles that are published at the same time as the training set, while test-future consists of samples that are created (or revised) after training and development sets. we further ensure that the subjects of biographies in both test sets are not in training or development sets. on average, each revision has 15.3 attributes in the infobox and 43.2 words in the first paragraph. details of data collection are included in appendix b. ## experiments and results for data-to-text generation on tempwikibio, we linearize the infobox and use it as the input to bart. common data-to-text metrics @xcite are used, including bleu-4 @xcite , meteor (lavie and agarwal, 2007), ter @xcite , and bertscore @xcite . for text-to-text summarization on xsum @xcite that consists of news articles and their corresponding summaries from bbc news, we report rouge scores @xcite and questeval @xcite , a qa-based faithfulness evaluation metric that checks if questions created from the summary can be addressed by reading the document with a qa model, and vice versa. the content transfer dataset (prabhumoye et al., 2019) considers sentences containing citations of news sources in wikipedia articles as the target for generation. many target sentences incorporate important dates from the cited articles, thus making it suitable to test our time-aware prompt design. to generate each target sentence, the context passage, which contains three sentences preceding the target sentence, and the cited news article are provided as input. rouge-l, meteor, and bertscore are computed for evaluation on the content transfer dataset. ## conclusion we study two types of time-aware prompts for injecting document timestamps into generation models. experiments on tempwikibio, our newly collected data-to-text generation dataset, and two text-to-text generation tasks show that linear prompts mostly enhance the processing of content other than dates for more informative and factual outputs. textual prompts build the association between the given temporal information and the generated temporal information, producing outputs with more factual dates.