Temporal Validity Change Prediction (2024)

Abstract: Temporal validity is an important property of text that has many downstream applications, such as recommender systems, conversational AI, and user status tracking. Existing benchmarking tasks often require models to identify the temporal validity duration of a single statement. However, many data sources contain additional context, such as successive sentences in a story or posts on a social media profile. This context may alter the duration for which the originally collected statement is expected to be valid. We propose Temporal Validity Change Prediction, a natural language processing task benchmarking the capability of machine learning models to detect context statements that induce such change. We create a dataset consisting of temporal target statements sourced from Twitter and crowdsource corresponding context statements. We then benchmark a set of transformer-based language models on our dataset. Finally, we experiment with a multitasking approach to improve the state-of-the-art performance.

## introduction information is not impervious to time. whether it be a post on a social media timeline like "i am going grocery shopping", a statement like "barack obama is the president of the united states" in a knowledge repository, or an advertisement like "ariana grande concert in town this weekend", sentences frequently contain inherently time-sensitive information. consequently, readers have to reason over whether the statement is still current and accurate when they ingest the information. this property of a statement can be described as its temporal validity @xcite @xcite . unlike event duration prediction @xcite @xcite , where the goal is to evaluate how long certain events take to unfold, temporal validity encompasses the contemporaneity and truthfulness of a statement at a given time. this means factors such as the statement creation time and the relative occurrence time of any information within it must also be considered. similar to previous work in the growing field of temporal commonsense reasoning @xcite , determining the temporal validity of a statement often relies on our prior commonsense understanding of the world due to underspecified temporal expressions. however, it can be difficult to accurately reason over this property when sentences actually belong to a larger content stream. for example, extracting a single post from a social media timeline or a single sentence from a book may remove important contextual information about the statement's validity period. the most frequently benchmarked task in the temporal validity domain requires models to estimate the duration for which a statement is valid in isolation @xcite , despite datasets often being created from streams of texts, such as wikihow articles or blog posts. to introduce the contextual dimension to such models, we propose a new natural language processing task called temporal validity change prediction (tvcp), visualized in figure 1 . this task is focused on identifying context statements that impact the duration for which a target statement is expected to be valid. tvcp can be used to benchmark or fine-tune temporal commonsense reasoning in machine learning models, an area in which even recent large language models (llms) still underperform @xcite . tvcp may also be linked to domains that are dependent on implicit temporal knowledge, such as narrative-, temporal information-, and temporal relation extraction @xcite @xcite . moreover, it provides an additional signal for duration-based temporal validity classifiers and can be used to identify temporal context statements in future data collection processes for various tasks related to temporal validity. in this sense, tvcp is a strong driver for future research into context-aware temporal validity reasoning. finally, tvcp-based classifiers could also improve model performance on various downstream tasks, such as the following. content prioritization: social media services like twitter rely on recommender systems to prioritize the vast amount of content that their users produce. for certain content, such as time-sensitive breaking news, estimating its importance through engagement metrics may not sufficiently reflect its urgency. by considering the temporal validity @xcite , it is possible to better estimate which content should be urgently shown to users. tvcp can be used to leverage a stream of social media posts as possible context to better estimate the temporal validity duration of any previously observed post. user status tracking: similarly, the content of a user's posts on social media could be utilized for other analytical or business purposes, such as predicting revenue streams @xcite @xcite or identifying trends in a community's or an individual user's behaviour @xcite @xcite . tvcp can identify posts that refer to previously observed time-sensitive information and detect chains of thought about topics that may not be self-contained. conversational ai: foundation models, such as chatgpt @xcite and bard @xcite , could incorporate the temporal validity of statements provided by the user to keep track of knowledge that is still relevant to the con-versation. using tvcp, new messages can provide context for previously learned time-sensitive facts. while we focus on consumer-driven downstream tasks, tvcp also generalizes to more structured language and statements, such as in knowledge repositories, news articles, or scientific literature. our main contributions are the following: ## task in the context of the previously described related tasks, we propose temporal validity change prediction (tvcp). similar to tvr, we require s t and s f as an input for classification and determine a ternary label that provides information about the impact of s f on s t . however, while tvr can be considered a standalone temporal validity reasoning process, tvcp primarily provides an additional signal for estimating the temporal validity duration of a statement, and is best used in conjunction with tvdp and a downstream task format where explicit temporal anchors for target statements can be derived. formally, we define tvcp in equation 4 (@xmath0, @xmath1, @xmath2, where tvdp c (s) is the temporal validity duration of a statement s given context c. figure 2 shows an example comparing the three temporal validity tasks. of note are the implicit semantic roles of s t and s f . while s f acts as additional contextual information, any information that is newly introduced in s f should not be evaluated on its temporal validity. the goal of the task is exclusively to estimate the change in the temporal validity duration of s t . we find that temporal validity change generally occurs along two dimensions. the first dimension is implicit versus explicit change. for example, an appointment mentioned in a target statement may be declared postponed in the follow-up statement, which would be an explicit change. on the other hand, the author may instead note in a followup statement that the appointment is for a surgery, which may cause us to re-evaluate the duration of even when a context statement supports that an action is still ongoing (tvr), that same context may decrease the expected temporal validity duration (tvcp). ## dataset we create a dataset for training and benchmarking tvcp, where each sample is a @xmath3,s f ,tvdp(st),tvdp s f (st),tvcp(st,s f )>. we quantify three types of time-sensitive information. actions (e.g., "i am baking bread"), events (e.g., "job interview tomorrow"), and temporary states (e.g.,"it is nice out today"). we analyse the time-sensitive information in 100 target statements and visualize the result in figure 4 . notably, onethird of sampled statements contain at least two distinct pieces of time-sensitive information with differing temporal validity spans, a trait that further distinguishes temporal validity research from event duration prediction. as previous work often focuses only on actions @xcite @xcite , our dataset may contain more diverse temporal information in comparison. for each target statement s t , we ask two crowdworkers to estimate tvdp(s t ) from the logarithmic class design shown in equation 5, which models uncertainty seen in humans when pinpointing distant events @xcite @xcite . if the annotators disagreed on the label, we supplied a third vote. we discarded any tweets that were annotated as no time-sensitive information, less than one minute or more than one month, and tweets where no majority agreement could be reached. third vote accepted rejected no 867 571 yes 1,012 546 for each target statement, we provide one sample per tvcp class, thus ensuring a balanced dataset. each sample consists of distinct s f , tvdp s f (s t ), and tvcp(s t , s f ) labels. we collect 5,055 samples from 1,685 target statements, noting average word counts for s t (16.1, σ = 4.40) and s f (14.6, σ = 3.29). similar to @xcite , our crowdsourced context statements are on average shorter than the target statements and have lower variance, likely due to participants aiming to replicate the form of the provided samples. in figure 6 , we plot the temporal validity change delta, which is the class distance between the original and the updated tvdp estimate. we find that, in most cases, the temporal validity duration of a target statement is shifted into the adjacent class. crowdsourcing was conducted on amazon mechanical turk. we split the crowdsourcing work into two distinct tasks: the annotation of target statements with a temporal validity duration estimate (tvdp), and the subsequent creation and annotation of context statements. working on either task required participants to fulfil certain criteria, such as a long track record of previous work on the platform with a high acceptance rate. in addition, each task required participants to pass a multiple-choice qualification test. finally, results were manually vetted per participant up to a certain threshold to ensure that participants were properly following the task descriptions. the full task setup for both crowdsourcing tasks is described in appendix b. other than the annotations, the only data we collect are sample counts and work times by each pseudonymized participant. we do not collect any personal or identifying data. we publish the resulting dataset for public use under the cc by 4.0 licence @xcite . in accordance with the twitter developer policy foot_0 , we only publish the tweet ids of sourced statements. ## conclusion and future work in this work, we have introduced tvcp, a task designed to reason over the impact of context on a statement's temporal validity duration. while tvcp is mainly intended as a driver for future work, we propose downstream applications by leveraging its output as a classification signal for tvdp. we provide a benchmark dataset for our task, as well as a set of baseline evaluation results. we find that the performance of fine-tuned classifiers can be consistently improved by explicitly incorporating tvdp labels as a loss signal. despite the impressive feats performed by foundation models, we report, similar to previous work @xcite , poor performance in the temporal commonsense domain. these findings show that users should carefully evaluate whether an llm properly understands a given task before choosing it over smaller, fine-tuned models. we also show that models pre-fine-tuned on existing temporal commonsense tasks do not necessarily lead to better performance on tvcp. finally, we highlight several advanced concepts of temporal validity that may be difficult to reason over for both lms and human annotators. future work could involve using tvcp-based classifiers to collect a larger number of temporal context statements. a comparison of context-aware tvdp classifiers with previous models @xcite could emphasize the importance of accurate semantic segmentation between targetand context statements. further, the use of our dataset for generative approaches could be explored, for example, in the context of generative adversarial networks. currently, we only evaluate the impact of single context statements on a specific target statement. in the future, longitudinal timeline modelling, as applied in moc, could be appropriated to reason over larger sections of a user's posting history. for our multitasking implementation, directions for future work could be changes to hyperparameters such as the weight of the auxiliary loss, changes to the definition of the auxiliary task (e.g., log-scaled regression or ordinal classification), or even entirely new auxiliary tasks. in the realm of llms, further experiments with different zero-shot or few-shot prompting strategies are also feasible. finally, research into models differentiating temporal and stationary information could significantly enhance the development of future temporal validity research.