Time-Aware Language Modeling for Historical Text Dating (2023)

Abstract: Automatic text dating(ATD) is a challenging task since explicit temporal mentions usually do not appear in texts. Existing state-of-the-art approaches learn word representations via language models, whereas most of them ignore diachronic change of words, which may affect the efforts of text modeling. Meanwhile, few of them consider text modeling for long diachronic documents. In this paper, we present a time-aware language model named TALM, to learn temporal word representations by transferring language models of general domains to those of time-specific ones. We also build a hierarchical modeling approach to represent diachronic documents by encoding them with temporal word representations. Experiments on a Chinese diachronic corpus show that our model effectively captures implicit temporal information of words, and outperforms state-of-the-art approaches in historical text dating as well.

## introduction the temporal dimension texts is critical to many natural language processing(nlp) tasks, such as information retrieval @xcite , question answering @xcite ), text summarization @xcite , event detection @xcite , and sentiment analysis @xcite . timestamps of documents provide essential clues for understanding and reasoning in time-sensitive tasks, whereas they are not always available @xcite . one way to solve this problem is to predict when a document was written according to its content, which is also known as automatic text dating @xcite . text dating has been widely introduced in computational sociology and digital humanities studies. one typical application of it is to date his- * *corresponding author. torical documents for the construction of digital libraries @xcite . such task is also called historical text dating @xcite , diachronic text evaluation @xcite , or period classification @xcite . compared to other dating tasks, historical text dating is more challenging as explicit temporal mentions(e.g., time expressions) that help to determine the written date of a document usually do not appear in it @xcite . to solve it, current research on historical text dating focuses on document modeling, trying to find the relationship between time and linguistic features @xcite . there are two main issues to historical text modeling. one is to learn word representations by diachronic documents. current research on word representation either learn static word embedding throughout the corpus @xcite , or learn dynamic word representations using pre-trained models @xcite . however, neither of them takes into account the relation between time and word meaning. for example, broadcast usually refers to sowing seeds before the 20th century; after that, it means transmitting by radios or tvs in most cases. studies on language evolution help to find the relationship between the same words in different time periods, since they often discuss them by mapping them into a same semantic space @xcite . however, how to apply such methods into document modeling for historical text dating is still unexplored. another is document modeling for historical texts. initial work on neural networkbased document modeling employ convolutional neural networks(cnn) or recurrent neural networks(rnn) @xcite , while recent research turns to pre-trained models like bert @xcite or roberta @xcite . however, these studies always treat time as a prediction target, but not a variable in modeling, which does not help to capture the temporal characteristics of diachronic documents. in fact, evidences from the research on language modeling show that time-based models help to capture semantic change of language, and consequently improve the performance of timesensitive tasks @xcite @xcite . to this end, some studies attempt to incorporate the time factor into language modeling, e.g., learning explicit temporal expressions via language models @xcite . however, as mentioned above, these methods are not suitable for historical text dating due to the lacking of explicit time expressions in historical texts. in this paper, we present a time-aware language model named talm, trying to introduce the time factor into the modeling procedure for historical text dating. inspired by the work in language evolution, we propose to learn word representations over documents of different time periods separately. in this way, each word has time-specific variants to the vocabulary. it is important because a document should be represented by word embeddings that are temporally consistent with it. we also apply an alignment approach to map all the time-specific variants of a word into the same semantic space in order to make them comparable. in particular, we propose temporal adaptation, a representation learning approach to learn word representations having consistent time periods with documents they belong to. this approach attempts to learn temporal word representations based on the knowledge of two aspects: time-specific variants of words and their contexts, which depict the temporal attribute of words from multiple perspectives. we also build a hierarchical model for long document modeling, where the temporal adaptation is applied for word representation. we validate our model on a largescale chinese diachronic corpus and an english diachronic corpus. experimental results show that our model effectively captures implicit temporal information of words, and outperforms state-of-theart approaches in historical text dating as well. our contributions can be summarized as follows: ## related work automatic text dating follows the research roadmap from traditional machine learning to deep learning technologies, like many other nlp tasks. early studies employ features by manual work to recognize temporal expressions within documents @xcite @xcite , which suffer from the problem of generalization and coverage rate. traditional machine learning methods focus on statistical features and learning models, such as naïve bayes @xcite , svm @xcite and random forests @xcite . recent studies turn to deep learning methods, and the experiments show their superior performances compared to traditional machine learning ones @xcite @xcite . pre-trained models are also leveraged to represent texts for the dating task, such as sentence-bert @xcite and roberta @xcite . pre-trained models show state-of-the-art performances on the text dating task; however, few of them consider the time attribute of words. language evolution studies explore the issue by modeling words from different time periods. such work can be categorized into three classes: 1)learning word embeddings for each time period separately, then mapping them into the same space via alignment methods @xcite ; 2) learning word embeddings for a time period first, then using them as the initial values to train word embedding for other time periods @xcite ; 3) learning unified word embeddings by introducing temporal variables into the learning procedure @xcite . however, static word embed-ding has the drawback of dealing with polysemous words @xcite , which may not be suitable for temporal word representation learning. in addition, there are still few studies on the language evolution modeling for historical text dating. on the other hand, research on the temporal pretrained models shows that encoding the temporal information in language modeling is beneficial in both upstream and downstream tasks @xcite . efforts have been made to build models that can learn temporal information, such as incorporating timestamps of texts into the input of the model @xcite , prefixing the input with a temporal representation @xcite , or learning the masked temporal information @xcite . however, there is still insufficient discussion of document modeling without explicit temporal mentions. ## the approach in this section, we explain the mechanism of the time-aware language modeling approach based on temporal adaptation, and how it is applied to the task of historical text dating. figure 1 shows the overall architecture of the proposed model, which consists of three main modules: word representation learning and alignment, temporal adaptation, and diachronic document modeling. ## experiments this section shows the experimental results as well as a discussion of the proposed model. specifically, we give the details about the experimental settings first, then we introduce the datasets and the preprocessing methods for them. in subsection 4.3, we show the results of the comparative experiments, the ablation tests and the parameter evaluations. finally, a discussion of our model is given. ## conclusion in this paper, we propose to incorporate the time attribute into the language modeling for histori-cal text dating by building talm, a time-aware language model. to address the problem of the model suffering from missing temporal expressions in historical texts, we propose a temporal adaptation model that makes the model have the ability of time-awareness. we also propose a hierarchical modeling method to incorporate the temporal adaptation and long document modeling. we conduct experiments using different attention mechanisms, showing the effectiveness of integrating temporal information on historical text dating task. our study provides new evidences to support the viewpoint that encoding temporal information contributes to improving the performances of language modeling.