Using Two Losses and Two Datasets Simultaneously to Improve T empo W i C Accuracy (2022)

Abstract: WSD (Word Sense Disambiguation) is the task of identifying which sense of a word is meant in a sentence or other segment of text. Researchers have worked on this task (e.g. Pustejovsky, 2002) for years but it’s still a challenging one even for SOTA (state-of-the-art) LMs (language models). The new dataset, TempoWiC introduced by Loureiro et al. (2022b) focuses on the fact that words change over time. Their best baseline achieves 70.33% macro-F1. In this work, we use two different losses simultaneously. We also improve our model by using another similar dataset to generalize better. Our best configuration beats their best baseline by 4.23%.

## introduction in 2019, pilehvar and camacho-collados (2018) introduced wic dataset. it is framed as a binary classification task between pairs of sentences including one identical target word with different meanings. in 2020, xl-wic was introduced by @xcite and made wic richer by providing more examples and adding more languages. we benefit from the english part of xl-wic as a helping dataset to improve the generalization of our model. @xcite baselines include: roberta @xcite base and large, timelms @xcite @xcite base and large. they examine two different methods of using these models: finetuning and sp-wsd layer pooling weights as explained in @xcite . the best result is for timelms-2019-90m with sp-wsd with 70.33% macro-f1. we examine roberta-base and timelms-jun2022-153m. for classification, many previous works (e.g. peters et al., 2019) use standard practice and con-catenate both sentences with a [sep] token and fine-tune the [cls] embedding. in this work, we use two different losses simultaneously, cross entropy loss on roberta classification head output as standard practice and add cosine embedding loss on average of target word output embeddings. ## experiments we have multiple configurations to test: ## future work in the future work, more configurations can be explored: ## conclusion in this work, we beat the best baseline of @xcite by a large margin. to do this we use two losses simultaneously (standard classifier head cross entropy loss and cosine embedding loss on average of target word output embeddings) to train sota lms, and also use xl-wic as a helping dataset to generalize better. the best lm was timelms-jun2022-153m which is a pre-trained model on 153m tweets.