Using Deep Mixture-of-Experts to Detect Word Meaning Shift for T empo W i C (2022)

Abstract: This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.

## introduction lexical semantic change (lsc) detection has drawn increasing attention in the past years @xcite . existed research works @xcite have shown that contextual word embeddings such as those produced by bert @xcite have great advantages over non-contextual embeddings for inferring semantic shift when there is limited data. meanwhile, many datasets are released to accelerate research in this direction. @xcite proposed word-in-context (wic) dataset as an benchmark for generic evaluation of context-sensitive representations. @xcite extended wic to xl-wic dataset with multilingual extensions. in contrast to these, tem-powic @xcite is crucially designed around the time-sensitive meaning shift and instances of word usage tied to twitter trending topics. our main work is to build a system that can detect semantic changes of target words in tweet pairs during different time periods for tempowic. it is framed as a binary classification task that addresses whether two instances of a target word have the same meaning. and pre-trained language models are adopted to produce contextual embeddings. ## system overview in this section, we first present the framework details for the models adopted in our work. then we introduce several strategies for improving the models' robustness. finally, we talk about the design of the model ensemble method. ## conclusion in this work, we provide an overview of the combined approach to detect the meaning shift in social media. we investigate the impact of adopting different pre-trained lms, finding that deberta performs best for this task. experimental results show that strategies such as data augmentation and adversarial training can enhance the model's robustness. in particular, incorporating pos information and word-level semantic representation with moe models can significantly improve performance. for future work, we will investigate how to incorporate different timelms with moe models for this task.