T empo W i C : An Evaluation Benchmark for Detecting Meaning Shift in Social Media (2022)

Abstract: Language evolves over time, and word meaning changes accordingly. This is especially true in social media, since its dynamic nature leads to faster semantic shifts, making it challenging for NLP models to deal with new content and trends. However, the number of datasets and models that specifically address the dynamic nature of these social platforms is scarce. To bridge this gap, we present TempoWiC, a new benchmark especially aimed at accelerating research in social media-based meaning shift. Our results show that TempoWiC is a challenging benchmark, even for recently-released language models specialized in social media.

## introduction one of the most studied challenges in nlp is lexical ambiguity. solutions include word sense disambiguation @xcite or entity linking @xcite , where words are linked to sense inventories such as wordnet @xcite or wikipedia. recently, notable progress has been made with the advent of language models (lms) and contextualized embeddings, crucially well equipped for modeling meaning in context @xcite . one notable limitation with current lexical semantics benchmarks, however, is that they are typically clean and time-invariant, where standard grammar is the norm, and have little to no account of language usage in real-world platforms like social media. however, there is ample agreement that modeling changes in language and topic distributions is crucial for modern nlp @xcite . thus, there is a rich body of literature concerned with, e.g., adapting existing word representations (mainly word embeddings) diachronically @xcite @xcite , exposing lms to time-specific data @xcite ., * annotation team. despite the above, few works have attempted to model the connection between meaning shift and social media. @xcite showed that trending words are a meaningful signal for predicting meaning shifts. this platform-specific insight is the main basis for the construction of our dataset. tempowic follows the simple formulation from the superglue word-in-context (wic) challenge (pilehvar and camacho-collados, 2019), which is particularly well suited for temporal meaning shift evaluation given that it is not reliant on a reference sense inventory. this change of paradigm has seen wide adoption, with multilingual extensions such as xl-wic @xcite , am2ico @xcite or mcl-wic @xcite , or reformulations such as wic-tsv @xcite . in contrast to these, tempowic is crucially designed around meaning shift and instances of word usage tied to twitter trending topics. as an additional contribution, along with the benchmark, we provide a set of robust baselines and analyses that highlight the challenging nature of the task. ## tempowic: temporal word in context in this section, we describe our process to build our evaluation benchmark for detecting meaning shift in social media. the task is framed as a simple binary classification problem in which a target word is present in two texts (tweets) posted during different time periods. the goal is to decide whether the meaning corresponding to the first target word in context is the same as the second one or not. ## evaluation in this section, we report baseline results on tem-powic using two different approaches which have proven successful on the wic task that inspired this work. more concretely, we report results based on pretrained lms using fine-tuning on the tweet pair as well as comparing the similarity of contextual embeddings. 4 evaluation metrics. the results are reported according to the standard macro-f1 metric for multiclass classification problems. accuracy is also reported for completeness but, given the unbalanced nature of the dataset, macro-f1 should provide a more accurate representation of the performance. ## conclusion this work introduced a new lexical semantics task and a dataset, tempowic, focused on meaning shift detection in twitter. while meaning representation is at the core of the task, the challenges of this task go beyond simple word sense disambiguation with a focus on its temporal aspect. to make the task realistic, we extracted twitter trending words for different periods and paired them with tweets from past periods. this makes the task more challenging and grounded in real-world applications for social media. we performed extensive experiments with standard meaning representation approaches based on language models. the results show that the task leaves ample room for improvement, with several avenues for future research on how to better integrate time-aware social media models with meaning representation techniques. the tempowic dataset and baseline scripts are available at github.com/cardiffnlp/tempowic.