MLL abs- LIG at T empo W i C 2022: A Generative Approach for Examining Temporal Meaning Shift (2022)

Abstract: In this paper, we present our system for the EvoNLP 2022 shared task Temporal Meaning Shift (TempoWiC). Different from the typically used discriminative model, we propose a generative approach based on pre-trained generation models. The basic architecture of our system is a seq2seq model where the input sequence consists of two documents followed by a question asking whether the meaning of target word changed or not, the target output sequence is a declarative sentence describing the meaning of target word changed or not. The experimental results on TempoWiC test set show that our best system (with time information) obtained an accuracy and Marco F-1 score of 68.09% and 62.59% respectively, which ranked 12th among all submitted systems. The results have shown the plausibility of using generation model for WiC tasks, meanwhile also indicate there’s still room for further improvement.

## introduction the evonlp shared task temporal meaning shift (tempowic) @xcite aims to judge whether the meaning of a target word in a pair of sentences (two tweets in this case) change or not. different from original wic (wordin-context) task, tempowic takes into account the temporal information in the text, as tweets in each pair are with the date when they are posted. therefore, that brings new challenges into this task -how to make use of the temporal information in the tweets? unlike conventional approaches for wic tasks that typically adopt discriminative models (such as bert or roberta) with an input consisting of a pair of sentences @xcite @xcite , here we propose a generative approach. in our approach we treat the temporal information in tweets as normal words * these authors contributed equally to this work. without complicated designation for task schema, aiming at exploiting the natural language understanding (nlu) capability of the pre-trained generation model @xcite . moreover, that could potentially inspire further developments for such tasks based on large language models (llms) such as gpt-3 @xcite with prompt learning which has been the focus of the community recently due to its superior zero-shot performance @xcite . specifically, for the input sequence we concatenate two tweets which are followed by a question asking whether the meaning of the target word change or not. the output sequence is a declarative sentence stating whether the meaning of the target word changed or not in the two tweets. in our approach, tempowic is framed as a generative qa task and the construction of the data follows the manner of template-based question generation @xcite @xcite . with the training data constructed in such format, we finetune the pre-trained generation models as a seq2seq model using a vanilla autoregressive generation objective @xcite @xcite . we submitted two systems (one with time information and the other one without time information) to tempowic for evaluation. results show that our best best system (with time information) on the tempowic test set obtained an accuracy and macro-f1 score of 68.09% and 62.59% respectively, which slightly outperforms the other system without time information with an accuracy and macro-f1 score of 68.02% and 61.14%. based on the evaluation results on tempowic test set, we found that pre-trained generation models are capable of capturing the meaning shift of target word in context. besides, results also show that time information (date of each tweet) can provide further improvement for performance, showing the importance of temporal information. in the rest of this paper, we will introduce the architecture of our system and give more detailed experimental results. ## methodology in this section, we briefly introduce how tem-powic task is formulated in this paper and how to train our system. ## conclusion in this paper, we present a generative approach based on pre-trained language models for the tem-powic task. experimental results show the plausibility of using pre-trained generative model for the tempowic task, which could potentially inspire further developments based on more pre-trained models.