(Chat) GPT v BERT Dawn of Justice for Semantic Change Detection (2024)

Abstract: In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.

## introduction lexical semantic change is the linguistic phenomenon that denotes words changing their meanings over time @xcite . an example is the word gay that changed from meaning cheerful to homosexual in the last century. this change is crucial to our understanding of historical texts. a nuanced grasp of semantic variation between groups and genre, and semantic change across time allows us to study languages, cultures, and societies through digitized text and opens up a range of research applications. computational approaches to semantic change are thus tools with immense potential for a range of research fields @xcite @xcite . not only can they broaden the field of historical linguistics and simplify lexicography, but they can also be fruitfully applied in the fields of sociology, history, and other text-based research. for instance, the computational modeling of semantic change is equally relevant when studying out-of-domain texts where language differs from the general language, like in medical @xcite and olfactory @xcite domains. the recent introduction of transformerbased @xcite language models (lms) has led to significant advances in natural language processing (nlp). these advances are exemplified in pretrained foundation models like bert @xcite and gpt, which "are regarded as the foundation for various downstream tasks" @xcite . bert has experienced a surge in popularity over the last few years, and the family of bert models has repeatedly provided state-of-the-art (sota) results for computational modeling of semantic change @xcite . however, research focus is now shifting toward chatgpt due to its impressive ability to generate fluent and high-quality responses to human queries, making it the fastest-growing ai tool. several recent research studies have assessed the language capabilities of chatgpt by using a wide range of prompts to solve popular nlp tasks @xcite . however, current evaluations generally (a) overlook the fact that the output of chatgpt is nondeterministic, foot_0 (b) rely only on contemporary and synchronic text, and (c) consider predictions generated by the chatgpt foot_1 web interface, which is based on the chat version of the gpt foundation model. as a result, these evaluations provide valuable insights into the generative, pragmatic, and semantic capabilities of chatgpt @xcite , but fall short when figure 1 : the title of this paper draws inspiration by the movie batman v superman: dawn of justice. we leverage the analogy of (chat)gpt and bert, powerful and popular lms, as two lexical superheroes often erroneously associated for solving similar problems. our aim is to shed lights on the potential of (chat)gpt for semantic change detection. in this paper, we propose to evaluate the use of both chatgpt and gpt -i.e., (chat)gpt foot_2 -to recognize (lexical) semantic change. our goal is not to comprehensively evaluate (chat)gpt in dealing with semantic change but rather to evaluate its potential as off-the-shelf model with a reasonable prompts from a human point of view, which may not necessarily be optimized for the model. recently, a novel evaluation task in nlp, called lexical semantic change (lsc), has been introduced as a shared task at semeval @xcite . the lsc task involves considering all occurrences (potentially several thousands) of a set of target words to assess their change in meaning within a diachronic corpus. as a result, this setup is currently not suitable for evaluating a gpt model, due to the limited size of its prompts and answers, as well as accessibility limitations such as an hourly character limit and economic constraints. in light of these considerations, we chose to evaluate the potential of (chat)gpt through the word-in-context (wic, pilehvar and camacho-collados, 2019) task, which has recently demonstrated a robust connection with lsc @xcite . in particular, we consider two diachronic extensions of the original wic setting, namely temporal wic (tempowic, @xcite and historical wic (histowic). our goal is to determine whether a word carries the same meaning in two different contexts of different time periods, or conversely, whether those contexts exemplify a semantic change. while tempowic has been designed to evaluate lms ability to detect short-term changes in social media, histowic is our adaptation of the semeval benchmark of historical text to a wic task for evaluating lms ability to detect long-term changes in historical corpora. considering the remarkable performance of contextualized bert models in addressing wic and lsc tasks @xcite @xcite , we compare the performance of (chat)gpt in tempowic and histowic to those obtained using bert. while bert is specifically designed to understand the meaning of words in context, (chat)gpt is designed to generate fluent and coherent text. through these two lexical superheros (see figure 1 ), we aim to illuminate the potential of (chat)gpt as off-the-shelf model and mark the dawn of a new era by assessing whether it already makes the approaches to wic and lsc, which rely on bert-embedding similarities, outdated. ## related work the significant attention garnered by chatgpt has led to a large number of studies being published immediately after its release. early studies mainly focused on exploring the benefits and risks associated with using chatgpt in expert fields such as education (lund and wang, 2023), medicine @xcite , or business @xcite . evaluation studies are currently emerging for assessing (chat)gpt's generative and linguistic capabilities across a wide range of downstream tasks in both monolingual and multilingual setups @xcite @xcite . most evaluations focus on chatgpt and involve a limited number of instances (e.g., 50) for each task considered @xcite @xcite . when the official api is used to query the gpt foundation model, this limit is imposed by the hourly token processing limit foot_3 and the associated costs. 5 when the web interface is used instead of the api, the limit is due to the time-consuming process of interacting with chatgpt that keeps humans in the loop. thus far, even systematic and comprehensive evaluations @xcite rely on repetition of a single experiment for each task. however, while individual experiments provide valuable insights into (chat)gpt's capabilities, they fall short in assessing the potential of (chat)gpt to solve specific tasks given its nondeterministic nature. multiple experiments need to be conducted to validate its performance on each task. in addition, current evaluations generally leverage tasks that overlook the temporal dimension of text, leaving a gap in our understanding of (chat)gpt's ability to handle diachronic and historical text. ## semantic change detection our evaluation relies on two diachronic definitions of the conventional word-in-context (wic) task, namely tempowic and histowic. wic is framed as a binary classification problem, where each instance is associated with a target word w, either a verb or a noun, for which two contexts, c 1 and c 2 , are provided. the task is to identify whether the occurrences of w in c 1 and c 2 correspond to the same meaning or not. both tempowic and histowic rely on the same definition of the task, while being specifically designed for semantic change detection in diachronic text. ## experimental setup in the following, we present our research questions (rqs) and the various setups we considered in our work. in our experiments, we evaluated the performance of (chat)gpt 3.5 over the tempowic and histowic test sets using both the official ope-nai api (gtp api) 7 and the web interface (chat-gpt web). 8 of the gpt 3.5 models available through the api, we assessed the performance of gpt-3.5-turbo. following @xcite , we employed the macro-f1 for multiclass classification problems as evaluation metric. ## experimental results in this section, we report the results of our experiments, while discussing the findings in regard to each research question. 10 rq1: (chat)gpt consistently followed our template in nearly all cases, thereby allowing us to evaluate its answers without human intervention. for gpt api, however, we noticed that the higher the temperature, the larger the tendency for deviations from the expected response format (see figure 2 ). chatgpt web only once answered with an incorrect format. to ensure impartiality, we classified the few (chat)gpt responses that did not adhere to the required format as incorrect answers. rq2: figure 3 shows the rolling average of the performance of gpt api across different temperatures, prompting strategies, and wic tasks. by using a window size of 4, we were able to consider 8 different experiments per temperature (for each temperature, we ran two experiments) 11 . figure 4 shows the performance of chatgpt web across different prompting strategies and wic tasks. figure 3 and 4 show that zsp consistently outperforms fsp on histowic. by contrast, fsp consistently outperforms zsp in tempowic when the gpt api is used. this result suggests that the incontext learning capability of gpt is more limited for historical data. in figure 4 , chatgpt web's performance with zsp outperforms that obtained with fsp for both tempowic and histowic, although the discrepancy is smaller. rq3: figures 3 and 4 show that (chat)gpt's performance on tempowic is consistently lower than its performance on histowic. in particular, in our experiments we observe that (chat)gpt's performance ranges from .551 to .689 on tempowic and from .552 to .765 on histowic. this suggests that (chat)gpt is significantly more effective for longterm change detection than for short-term change detection. we believe that this might involve word meanings that were not explicitly covered during training, potentially allowing (chat)gpt to detect anomalies from the usual patterns. we will further investigate this aspect in our future research. for the sake of comparison, we report sota performance in rq4: figure 3 shows that, on average, higher performance is associated with lower temperatures for both tempowic and histowic, with accuracy decreasing as temperature values increase. thus, we argue that high temperatures do not make it easier for gpt to solve wic tasks or identify semantic changes effectively. ## conclusion in this study, we empirically investigated the use of the current (chat)gpt 3.5 to detect semantic change. our goal is not to comprehensively evaluate (chat)gpt in dealing with semantic change, but rather to acknowledge its potential while also raising concerns and questions about its off-the-shelf use. in this regard, we used reasonable prompts from a human point of view, which may not necessarily be optimized for the model. we used the tempowic benchmark to assess (chat)gpt's ability to detect short-term semantic changes, and introduced a novel benchmark, histowic, to assess (chat)gpt's ability to recognize long-term changes. when considering the standard 12 layer of bert, our experiments show that (chat)gpt achieves comparable performance to bert (although slightly lower) in regard to detecting longterm changes, but performs significantly worse in regard to recognizing short-term changes. we find that bert's contextualized embeddings consistently provide a more effective and robust solution for capturing both short-and long-term changes in word meanings. there are two possible explanations for the discrepancy in (chat)gpt's performance between tempowic and histowic: i) histowic might involve word meanings not explicitly covered during training, potentially aiding (chat)gpt in detecting anomalies; ii) tempowic involves patterns typical of twitter (now x), such as abbreviations, mentions, or tags, which may render it more challenging than histowic. in light of our findings, we argue that (chat)gpt 3.5 might be the hero the world deserves but not the one it needs right now foot_11 , in particular for computationally modeling meaning over time, and by extension, for the study of semantic change. nevertheless, during the course of our research, updates to (chat)gpt became available and gained popularity, leading research and practitioners to conduct new experiments on these updated models. particularly noteworthy is a recent study by @xcite , which showcased remarkable performance on lsc using the gpt-4 model. inspired by this research, our ongoing and future work is focused on further exploring the capabilities of gpt-4 for modeling semantic change.