Temporal-Aware Soft Prompt Tuning for Automatic Text Dating (2025)

Abstract: This paper presents Temporal-aware Soft Prompt Tuning (TASPT), a novel approach for automatic text dating. Unlike existing methods, which often overlook the evolution of word meanings in texts spanning long periods, TASPT incorporates the unique characteristics of historical texts. It introduces a temporal-aware text representation that dynamically captures both semantic variance and invariance. This representation is combined with a soft prompt, enabling efficient parameter tuning for automatic text dating. Experiments show that TASPT outperforms all existing methods on two diachronic datasets: the Twenty-Four Histories and the Royal Society Corpus.

## introduction to enhance the performance of various natural language processing tasks, such as information retrieval, machine translation, and automatic question answering, accurately understanding the temporal information in texts is essential. many of these tasks are time-sensitive, as much factual information relies on the temporal context. for example, in chinese, the word "寺" (temple) primarily referred to government offices in ancient times but now typically denotes buddhist monasteries. consequently, different eras may yield distinct answers to the same question. however, in practical applications, not all texts can be clearly labeled with corresponding temporal information. one effective solution to this challenge is automatic text dating (atd). performing atd on historical texts with long time spans presents two main challenges. the first is learning accurate representations of word meanings. current atd methods mainly rely on static word embeddings @xcite or pre-trained language models @xcite @xcite to capture text features. however, as word meanings evolve over time, these static methods may fail to capture semantic differences across different periods, impacting atd performance. the second challenge is integrating temporal semantic information into the model. recent research indicates that continuing training of pre-trained language models (plms) on texts from different time periods can improve downstream task performance @xcite @xcite . however, these methods require separate models for each time period, hindering the dynamic integration of temporal information in downstream tasks. another research proposed talm @xcite , which uses a temporal alignment module to synchronize word representations across periods and a temporal adaptation module to incorporate features, but this pipeline approach is prone to cascaded error propagation. recently, llms excel in many natural language processing tasks by leveraging vast amounts of data. however, atd poses a unique challenge due to the limited atd-specific data in their training corpora and the risk of catastrophic forgetting. this limits llms' ability to grasp the temporal features crucial for atd. our experiments with open-source models highlight these challenges in applying llms to atd. to address the aforementioned challenges, we propose a novel model named temporal-aware soft prompt tuning (taspt). inspired by @xcite , our approach decomposes word meaning representation in historical texts into three key components: semantic variance, semantic invariance, and temporal context features. semantic variance captures changes in word meanings over time, semantic invariance identifies meanings that remain stable, and temporal context features highlight meanings across various contexts. we define specific learning objectives for each component during pre-training to construct a temporal-aware text representation tailored for the atd task. this representation is then integrated into soft prompts, and we apply parameter-efficient fine-tuning techniques to optimize the transformer network, achieving high-performance atd. our contributions are summarized as follows: ## methods in this section, we introduce a novel framework based on p-tuning that fine-tunes soft prompts using a frozen pre-trained language model, significantly reducing storage and memory requirements during atd training. however, since atd tasks require representations from different time periods, p-tuning alone is insufficient. to address this, we propose a temporal-aware text representation that projects texts from various historical periods into distinct vector spaces. by integrating this representation with soft prompts, our method better captures semantic shifts over time, enhancing atd system accuracy. the system architecture of taspt v1 lstm is shown in figure 1 . this section is organized as follows: section 3.1 outlines the structure of atd-bert, which includes three core subtasks: semantic variance learning (section 3.1.1), semantic invariance learning (section 3.1.2), and temporal context-aware learning (section 3.1.3). section 3.2 then explains how atd-bert's outputs are integrated into three distinct peft architectures. ## conclusion in this paper, we introduce temporal-aware soft prompt tuning (taspt), a novel method for automatic text dating. we design a temporal-aware text representation that captures semantic variance, invariance, and temporal contextual features for the atd task, integrating it into a soft prompt with efficient parameter fine-tuning to align with pre-trained language models. our experiments on chinese and english historical datasets show that taspt outperforms existing atd methods, large language models, and other soft prompt approaches. we also found that current large models struggle with atd, and future work will focus on improving their performance in this area.