Salient Span Masking for Temporal Understanding (2023)

Abstract: Salient Span Masking (SSM) has shown itself to be an effective strategy to improve closed-book question answering performance. SSM extends general masked language model pretraining by creating additional unsupervised training sentences that mask a single entity or date span, thus oversampling factual information. Despite the success of this paradigm, the span types and sampling strategies are relatively arbitrary and not widely studied for other tasks. Thus, we investigate SSM from the perspective of temporal tasks, where learning a good representation of various temporal expressions is important. To that end, we introduce Temporal Span Masking (TSM) intermediate training. First, we find that SSM alone improves the downstream performance on three temporal tasks by an avg. +5.8 points. Further, we are able to achieve additional improvements (avg. +0.29 points) by adding the TSM task. These comprise the new best reported results on the targeted tasks. Our analysis suggests that the effectiveness of SSM stems from the sentences chosen in the training data rather than the mask choice: sentences with entities frequently also contain temporal expressions. Nonetheless, the additional targeted spans of TSM can still improve performance, especially in a zero-shot context.

## introduction salient span masking (ssm), first introduced by @xcite for retrieval-based language modeling, has shown performance gains for closedbook question answering (cbqa) @xcite . ssm is a form of intermediate pretraining @xcite , where a pretrained model such as a bert @xcite or t5 @xcite is trained further before taskspecific finetuning, generally on more specialized data that does not require expensive annotations. specifically, ssm uses the masked language modeling objective but only masks named entities and dates in sentences from english wikipedia articles; these "salient" spans likely contain more facts, so the language model must memorize more facts in order to do the task successfully @xcite . the authors use a named entity recognition model to identify entity spans and a regular expression to identify date spans. while this works well for knowledge intensive downstream tasks, such as entity-centric question answering, it remains unclear whether it is helpful for tasks that are less aligned with the data, such as common sense or temporal reasoning. moreover, is it possible to select spans that are more related to a downstream task in order to get further performance gains? in this work, we investigate ssm for tasks that require understanding temporal expressions. while ssm does include dates, the tasks we investigate include other complex temporal expressions such as durations and intervals. to that end, we introduce temporal span masking (tsm): an intermediate pretraining strategy for predicting spans that are likely temporal expressions (figure 1 ). similar to ssm, tsm is automatically generated from english wikipedia articles. we compare models trained on tsm and ssm on three temporal tasks, namely mc-taco @xcite , timedial @xcite and situatedqa @xcite , and for one general-purpose question answering (qa) task of natural questions (nq) @xcite . we summarize our contributions as follows: ## methods following roberts et al. (2020), we utilize intermediate training to improve pretrained models' generalization to downstream tasks. all models are initialized from the encoder-decoder t5-1.1-xxl language model @xcite , which was shown to have the best closed-book qa performance in @xcite . ## results and discussion our main results can be found in t5 and t5-lm the t5 model sets a relatively high baseline compared to previously reported models. the t5-lm model's extra non-domain-specific pretraining does not help on any task, suggesting entities the entities model, which is trained on only non-temporal entity spans, performs better overall than the tsm task. it only does worse on the timedial dataset, which is almost entirely focused on conversational, non-knowledge based contexts. it still does substantially better than the base t5 model when no finetuning data is available. this high performance is possibly due to the prevalence of temporal spans in the ssm training data. running sutime on the entities data reveals that 45% of its training examples contain at least one date, duration, set, or time. this suggests that sentences with named entities in general already carry temporal-salient information useful for downstream temporal tasks. see appendix b for a full breakdown of the co-occurrences. ## conclusion in this work, we investigate ssm as it relates to temporal tasks that require understanding both commonsense and world knowledge questions and propose a new intermediate training method which selects spans generated by a temporal parser. these intermediate training strategies result in the best overall reported results on the selected downstream tasks. however, we find that even the entity spans from ssm are helpful for temporal tasks, likely because entity-containing examples also contain informative temporal knowledge.