Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation (2023)

Abstract: Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words. We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using time-sensitive templates. Given two snapshots C_1 and C_2 of a corpus taken respectively at two distinct timestamps T_1 and T_2 , we first propose an unsupervised method to select (a) pivot terms related to both C_1 and C_2 , and (b) anchor terms that are associated with a specific pivot term in each individual snapshot.We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms.Moreover, we propose an automatic method to learn time-sensitive templates from C_1 and C_2 , without requiring any human supervision.Next, we use the generated prompts to adapt a pretrained MLM to T_2 by fine-tuning using those prompts.Multiple experiments show that our proposed method significantly reduces the perplexity of test sentences in C_2 , outperforming the current state-of-the-art.

## introduction contextualised word embeddings produced by mlms @xcite @xcite represent the meaning of a word with respect to the context in which it appears in and have reported substantial performance gains in various nlp tasks. the usage of words change over time and the same word might be associated with different words to mean different concepts over time @xcite @xcite . for example, the word gay has gradually changed its meaning from happy to homosexual over the last five decades @xcite . however, mlms are often trained using a static snapshot of a corpus taken at a specific timestamp, and are not updated afterwards. because of this reason, existing pretrained mlms do not capture the temporal semantic variations of words. for example, @xcite showed that neither the original version of bert @xcite nor roberta @xcite are upto-date with the information related to the current coronavirus pandemic. to address the above-mentioned limitations, we propose a dynamic contextualised word embedding (dcwe) method that adapts a given pretrained mlm from one timestamp t 1 to another t 2 using two snapshots of a corpus c 1 and c 2 , sampled respectively at times t 1 and t 2 . we represent a word x by an embedding that depends both on the context c of x, as well as on time t . our word embeddings are dynamic because they depend on the time, and contextualised because they also depend on the context. we model the problem of adapting a given pretrained mlm to a specific timestamp t 2 as an instance of prompt-based fine-tuning @xcite , which has been successfully used in prior work to adapt mlms to various tasks such as relation representation @xcite , domain adaptation @xcite , natural language inference @xcite and question answering @xcite . compared to fine-tuning mlms on manually labelled training instances, which might not be readily available or costly to manually annotate in sufficient quantities for a particular task to fine-tune a largescale mlm, prompt-based methods require only a small number of prompts @xcite . luckily, in our case of temporal adaptation of mlms @xcite , such prompts could be generated from a handful of manually designed templates ( §3.1) or automatically extracted from unlabelled data ( §3.3). this aspect of our proposed method is particularly attractive compared to prior work (see § 2) on dwes @xcite @xcite that require retraining of mlms from scratch to incorporate the time-sensitive constraints into the embedding spaces. we first extract pivot words, w, that are common to both c 1 and c 2 . second, we extract anchor words u and v that are strongly associated with w in respectively c 1 and c 2 . we then propose several methods to score tuples @xcite such that the semantic variation of w from t 1 to t 2 is captured by its association with respectively u and v. finally, we generate a large number of textual prompts using the top-scoring tuples (w, u, v) according to each method to fill the slots in manually written templates such as "⟨w⟩ is associated with ⟨u⟩ in ⟨t 1 ⟩, whereas it is associated with ⟨v⟩ in ⟨t 2 ⟩." here, the slots corresponding to t 1 and t 2 are filled by specific years when respectively c 1 and c 2 were sampled. we differentiate templates from prompts throughout the paper where the latter is formed by filling one or more slots in the former. we further propose a method to automatically generate templates from sentences selected from c 1 and c 2 using a text-to-text transformation model @xcite , thereby obviating the need to manually create templates. finally, the given mlm is adapted to t 2 by fine-tuning it on the generated prompts. experimental results conducted on reddit, yelp, arxiv and ciao datasets show that the proposed prompt-based time-adapting of mlms consistently outperforms previously proposed dcwes @xcite and temporal adaptation methods @xcite reporting better (lower) perplexity scores on unseen test sentences in c 2 . the source code for our proposed method is publicly available. 1 ## related work methods that use part-of-speech @xcite , entropy @xcite , latent semantic analysis @xcite and temporal semantic indexing @xcite have been proposed for detecting changes in word meanings. in semeval-2020 task 1 @xcite two subtasks were proposed for detecting lexical semantic change: a binary classification task (for a given set of target words, decide which words had their meaning altered, and which ones not) and a ranking task (rank a set of target words according to their degree of lexical semantic change between the two corpora). @xcite showed that contextualised embeddings obtained from an 1 https://github.com/livnlp/timeadapted-dcwe mlm can be used to measure the change of word meaning. @xcite proposed a temporal attention mechanism by extending the self-attention mechanism in transformers, where time stamps of the documents are considered when computing the attention scores. @xcite proposed a method to predict semantic change of words by comparing the distributions of contextualised embeddings of the word between two given corpora, sampled at different points in time. our goal in this paper extends beyond the detection of a subset of words with a change in lexical semantics, and to adapt mlms over time. dwes @xcite @xcite incorporate extralinguistic information such as time, demographic or social aspects of words with linguistic information. @xcite learnt demographic word embeddings, covering attributes such as age, gender, location and religion. @xcite learnt socialised word embeddings considering both a social media user's personal characteristics of language use and that user's social relationships. however, @xcite showed that temporal factors have a stronger impact than socio-cultural factors when determining the semantic variations of words. consequently, in this paper we focus on the temporal adaptation of dcwes. diachronic language models that capture the meanings of words at a particular point in time have been trained using historical corpora @xcite . these prior work learn independent word embedding models from different corpora. this is problematic because information related to a word is not shared across different models resulting in inefficient learning, especially when word occurrences within a single snapshot of a corpus are too sparse to learn accurate embeddings. @xcite proposed a dynamic bernoulli embedding method based on exponential family embeddings, where each word is represented by a one-hot vector with dimensionality set to the vocabulary size. this model is extended to the temporal case by considering different timeslices where only the word embedding vector is time-specific and the context vectors are shared across the corpus and over time-slices. because the joint distribution over time and context is intractable, they maximise the pseudo log-likelihood of the conditional distribution for learning the parameters of their dwe model. @xcite proposed a domain adaptation method based on automatically learnt prompts. given a test example, they generate a unique prompt and conditioned on it, then predict labels for test examples. although their method uses prompts to adapt a model, they do not consider temporal adaptation of mlms, which is our focus. moreover, we do not require any labelled examples in our proposal. amba @xcite proposed a model updating method using vocabulary composition and data sampling to adapt language models to continuously evolving web content. however, their work is specific to one dataset and two classification tasks, and focuses on incremental training. @xcite introduced a benchmark for everevolving language models, utilising the difference between consecutive snapshots of datasets, to track language models' ability to retain existing knowledge while incorporating new knowledge at each time point. @xcite studied the lifelong language model pretraining problem, where the goal is to continually update pretrained language models using emerging data. @xcite introduced a diagnostic dataset to investigate language models for factual knowledge that changes over time and proposed an approach to jointly model texts with their timestamps. they also demonstrated that models trained with temporal context can be adapted to new data without retraining from scratch. @xcite proposed tempobert, where they insert a special time-related token to each sentence and fine-tune bert using a customised time masking. tempobert reports superior results in semeval 2020 task 1 semantic variation detection benchmark. as shown later in §4.1, our proposed method outperforms tempobert. @xcite proposed dcwes, which are computed in two stages. first, words are mapped to dynamic type-level representations considering temporal and social information. the typelevel representation of a word is formed by combining a non-dynamic embedding of a word and a dynamic offset that is specific to the social and temporal aspects of the word. second, these dynamic embeddings are converted to context-dependent tokenlevel representations. to the best of our knowledge, this is the only word embedding method that produces both dynamic as well as contextualised representations, thus mostly relates to us. as shown in § 4, our proposed method outperforms their dcwes on four datasets. ## prompt-based time adaptation given two snapshots c 1 and c 2 of a corpus taken respectively at timestamps t 1 and t 2 (> t 1 ), we consider the problem of adapting a pretrained mlm m from t 1 to t 2 . we refer to a word w that occurs in both c 1 and c 2 but has its meaning altered between the two snapshots as a pivot. we propose three methods for selecting tuples (w, u, v), where u is closely associated with the meaning of w in c 1 , whereas v is closely associated with the meaning of w in c 2 . we name u and v collectively as the anchors of w, representing its meaning at t 1 and t 2 . if the meaning of w has changed from t 1 to t 2 , it will be associated with different sets of anchors, otherwise by similar sets of anchors. we then fine-tune m on prompts generated by substituting (w, u, v) in templates created either manually ( §3.1) or automatically ( §3.3). ## experiments and results datasets: we use the following four datasets that were collected and used by @xcite for evaluating dcwes: yelp, reddit, arxiv, and ciao @xcite . details of these datasets and all pre-processing steps are detailed in appendix c. we remove duplicates as well as texts with less than 10 words in each dataset. we then randomly split each snapshot of a dataset into training, development, and test sets, containing respectively 70%, 10% and 20% of the original dataset. evaluation metric: if an mlm is correctly adapted to a timestamp t 2 , it should be able to assign higher probabilities to the masked out tokens in unseen texts in c 2 , sampled at t 2 . we follow prior work on dcwe @xcite and use the masked language modelling perplexity as our evaluation metric on test texts in c 2 . if an mlm is well-adapted to t 2 , it will have a lower perplexity for test texts in c 2 . baselines: to put our results into perspective, we consider the following baselines: original bert: we use pretrained bert-base-uncased 5 as the mlm without any fine-tuning to be consistent with @xcite . further evaluations on roberta @xcite are given in appendix b. bert(t 1 ): we fine-tune the original bert model on the training data sampled at t 1 . bert(t 2 ): we fine-tune the original bert model on the training data sampled at t 2 . note that this is the same training data that was used for selecting tuples in §3.2 ft: the bert models fine-tuned by the proposed method. we use the notation ft(model, template) to denote the model obtained by fine-tuning a given mlm using a template, which is either manuallywritten (manual) or automatically-generated (auto) as described in §3.3. hyperparameters: we use the held-out development data to tune all hyperparameters. we follow the recommendations of @xcite for fine-tuning bert on small datasets and use weight decay (0.01) with bias correction in adam optimiser @xcite . we use a batch size of 4, learning rate of 3×10 -8 , the number of tuples used for prompt-based fine-tuning (k) is selected @xmath0, 1000, 2000, 5000, 10000}, and the number of epochs is set to 20. (further details on hyperparameters are given in appendix d). we used a single nvidia rtx a6000 and 64 gb ram in our experiments. it takes approximately 6 hours to fine-tune, validate and test all methods reported in the paper for the four datasets. the run time varies depending on the number of tuples used in the proposed method. tuple selection takes, on average, 0.5 hours. ## conclusion we propose an unsupervised method to learn dcwes by time-adapting a pretrained mlm using prompts from manual and automatic templates. experimental results on multiple datasets demonstrate that our proposed method can obtain better perplexity scores on unseen target texts compared to prior work. in the future, we plan to extend the proposed method to adapt multilingual word embeddings. 6 www.burgerville.com 7 www.joespizzanyc.com 8 www.dominos.com 9 www.nokia.com 10 www.apple.com 11 www.blackberry.com