T empo F ormer: A Transformer for Temporally-aware Representations in Change Detection (2024)

Abstract: Dynamic representation learning plays a pivotal role in understanding the evolution of linguistic content over time. On this front both context and time dynamics as well as their interplay are of prime importance. Current approaches model context via pre-trained representations, which are typically temporally agnostic. Previous work on modelling context and temporal dynamics has used recurrent methods, which are slow and prone to overfitting. Here we introduce TempoFormer, the first task-agnostic transformer-based and temporally-aware model for dynamic representation learning. Our approach is jointly trained on inter and intra context dynamics and introduces a novel temporal variation of rotary positional embeddings. The architecture is flexible and can be used as the temporal representation foundation of other models or applied to different transformer-based architectures. We show new SOTA performance on three different real-time change detection tasks.

## introduction linguistic data sequences are generated continuously over time in the form of social media posts, written conversations or documents that keep evolving (e.g. through regular updates). while a large body of work has been devoted to assessing textual units or sub-sequences in isolation -i.e. in emotion classification @xcite , icd coding @xcite , task-specific dialogue generation @xcite , irony and sarcasm detection @xcite ) -such approaches leave significant historical (often timestamped) context unused. fig. 1 provides an example from the task of identifying mood changes through users' online content, where the last post in isolation cannot denote if there has been a 'switch' in the user's mood -the historical content provides important context for the user's originally positive mood, en-hancing the signal for a negative switch in their behaviour. figure 1 : paraphrased example from the task of identifying moments of change in individuals' mood @xcite . here, the historical content (light grey) provides important linguistic context towards identifying a switch, a sudden mood shift from positive to negative, in the user's behaviour at the last post (black). dynamic representation learning approaches aim to tackle this challenge. dynamic word embedding methods have been studied in the context of semantic change detection @xcite . while changes in this context occur over long time periods, dynamic representation learning has been explored in other more temporally fine-grained tasks such as event detection @xcite @xcite , fake news detection @xcite @xcite and mental health condition detection @xcite @xcite . such temporally fine-grained tasks significantly differ from semantic change detection approaches: not only on the temporal granularity aspect, but crucially with respect to event timeline length, irregularities in change frequency, annotation requirements and problem formulation. therefore the adaptation of methodologies between the various sets of temporal change detection categories is at best challenging. correspondingly, fine-grained dynamic representation learning research remains also largely task or even dataset specific. transformer-based injection. the above mentioned approaches have relied on either pre-trained contextualised representations or transformerbased model layers @xcite to fine-tune representations before feeding them into rnn and cnn-like architectures as so far they had been shown to outperform transformer-based models @xcite @xcite . however, since lstm-based systems tend to overfit small datasets, transformer-based methods that overcome this issue would be a preferable choice @xcite ). yet so far adapting layers on top of a transformer fails to strike the right balance between representation learning and task dynamics @xcite . temporal modelling. although integration of time in language models has been explored for temporal adaption @xcite in semantic change detection, @xcite there is not yet work that explores the abilities of transformers to model temporally distant textual sequences (streams). recently llms have been shown to fall short in terms of temporal reasoning @xcite , especially in event-event temporal reasoning @xcite . here we make the following contributions: ## methodology here we introduce the tempoformer architecture. we first provide the problem formulation ( §3.1), followed by model overview ( §3.2) and then discuss the various model components ( §3.3-3.7). ## conclusion we introduce tempoformer, a transformer-based model for change detection operating on textual (timestamped) streams. importantly we do so by avoiding recurrence, and only modifying the last two layers of the transformer. furthermore, tem-poformer has the ability to model the temporal distance between textual units through a modification of rotary positional embeddings. the model achieves new sota, outperforming recurrent and llm-based models on three different change detection tasks with datasets of varying temporal granularity and linguistic diversity, without loss in generalisability. we demonstrate its usability as a foundation model in other architectures, showing it strikes the right balance between word-level, post-level and stream-level linguistic and temporal dynamics. lastly, we showcase its flexibility in terms of base model integration, further boosting stream-level performance on par with post-level gains.