The Finer They Get: Combining Fine-Tuned Models For Better Semantic Change Detection (2023)

Abstract: In this work we investigate the hypothesis that enriching contextualized models using fine-tuning tasks can improve theircapacity to detect lexical semantic change (LSC). We include tasks aimed to capture both low-level linguistic information like part-of-speech tagging, as well as higher level (semantic) information. Through a series of analyses we demonstrate that certain combinations of fine-tuning tasks, like sentiment, syntactic information, and logical inference, bring large improvements to standard LSC models that are based only on standard language modeling. We test on the binary classification and ranking tasks of SemEval-2020 Task 1 and evaluate using both permutation tests and under transfer-learningscenarios.

## introduction the last few years have seen a growing interest in language change, specifically in lexical semantic change (lsc), from the nlp community. lsc is the linguistic phenomenon in which words' meaning may change over time (for example by adding senses, or broadening/narrowing in their meaning scope). originally, the study of lexical semantic change aspired to understand the phenomenon from a linguistic perspective @xcite @xcite . however, it was also motivated by the need for better handling of semantic change in other text-based research disciplines that work with historical texts (e.g., lexicographers, historians). in more recent times, the understanding that general purpose nlp models also need to accommodate for the fluidity of word meaning has reached the greater nlp community, bringing with it the realization that lsc plays a vital role @xcite . it is particularly visible in the deterioration of model performance over time because the language on which models and other algorithms are (pre-)trained, starts to drift as time passes @xcite . when deployed, these models process text from time periods they were not trained on, which hinder their performance. this wide, multi-disciplinary interest in lsc, has driven the development of many computational models for language change @xcite . in addition, much work has been devoted to supporting this progress by curating evaluation datasets that provide appropriate testing of these new models. most of these datasets are in the form of semeval tasks and contain high quality, humanly annotated lists of words. each word has either changed its meaning between the considered time periods, or remained stable in meaning, and each list is accompanied by relevant historical corpora. this has become the de-facto evaluation standard in the field. when reviewing the different models that are evaluated in semeval, most of them use the same suit of methods that rely on standard distributional models of meaning. these models are either trained solely on historical text (e.g., sgns or other static models), or use contextualized models pre-trained on a large "general purpose" text. 1 all of these models generate meaning representations in vector form for words from a historical corpus and compare them to vectors representing the modern meaning of the same word. although the models differ (e.g., in terms of data or training objectives), they all share the same basic trait -they rely on meaning representation based on neighboring words without additional linguistic information. this state of affairs raises a question: do models that are based solely on collocation statistics and trained exclusively on a masked-word prediction task suffice for our purpose? that is, are classic distributional models able to capture the full repertoire of word meaning, and then access it when analyzing meaning change? in this work, we investigate several concerns that we think a-priori suggest that additional linguistic information is beneficial for unsupervised lexical semantic change detection. one direct path forward is fine-tuning pretrained models on additional tasks or domains like question answering and sentiment analysis. it has been demonstrated that fine-tuning of models helps even when the fine-tuned tasks are different than the target task (for example, fine-tuning on textual summarization and testing on question-answering) in a sort of a zero/few-shot transfer learning @xcite @xcite . therefore, it is reasonable to assume that enriching a contextualized model with additional fine-tuning tasks would lead to improved performance also for lsc detection. in this paper, we test this hypothesis with one of the top-performing models of lsc detection in english and explore the potential to improve it by enriching it through a set of fine-tuning tasks. we provide our code here. foot_1 2 related literature ## method our aim is to test whether a state-of-the-art method for detecting lexical semantic change, based on pre-trained contextual embeddings, can be improved by adding fine-tuned layers. therefore, we start with a bert model and detect semantic change following @xcite . we chose their model as it was the best-performing system in english in the postevaluation stage of semeval-2020 task 1. next, we add information from fine-tuning for tasks that go beyond a masked language model objective. we include tasks aimed to capture both low-level linguistic information like part-of-speech tagging, as well as higher-level (semantic) information such as sentiment analysis, linguistic inference, and machine reading comprehension. ## evaluation for our experiments, we use a standard evaluation dataset for lsc. ## experiments two models are used as baselines, relative to which we test if adding ft (either adaptors or local ft) improves performance. one of the baseline models was used as the basis on top of which the different ft combinations were tested. we choose the best combination of ft tasks by analyzing the results of the ranking task and then test it on the binary classification task using a modified decision criterion (i.e., roc analysis). because the two tasks are different, this allows us to use the latter as an ad-hoc evaluation test. ## discussion and conclusion in this paper, we investigate our hypothesis that adding linguistic information to pre-trained language models by means of fine-tuning can lead to improved performance on unsupervised lexical semantic change detection. we chose two classic lsc tasks, ranking and binary classification, from semeval-2020 task 1 @xcite . to simplify and speed up the process of fine-tuning we used adaptors @xcite , which are pre-trained fine-tuning modules that can augment existing contextualized models and are readily and freely available for english. first, we tried single ft tasks, which showed little or no improvement over the bert baseline. then we combined several ft tasks together by means of simple averaging and found considerable improvements. adding linguistic information, like part-ofspeech, to a standard (masked) language model can offer additional information that improves the ability of the models to detect lexical semantic change. however, some kinds of information are adjacent to semantic change and therefore make the models capture change, but not necessarily semantic change. in future work, we will conduct indepth analysis of the worst models, to see what information they capture instead and why this information seems to hurt the performance of the lsc model. our work is not the first to introduce linguistically augmented contextualized models for the task of lsc. @xcite used an ensemble method to inject linguistic information, and reported performance gains in lsc tasks. however, they focus on "low-level" morphosyntactic information. our approach, in addi-figure 1 : roc curves of the best 3 models in the ranking task as they perform on the classification task. left: prt, right: apd. see one deficiency of our results is that they are based on a small evaluation dataset, which means that the improvements we report could be attributed to chance (or model over-fitting). to mitigate this concern, and add scientific rigour to our analyses, all results were tested in permutation tests and are reported with their p-values. evaluation is also done with comparison to two strong baseline models, each of which provides different perspective to test our research hypothesis. outperforming the kutuzov and giulianelli (2020) model suggests that enriching models with additional linguistic information is highly beneficial, and outperforming histbert supports the idea that this information cannot be gained even when a model is exclusively (pre-) trained on historical text. overall our results clearly suggest that lsc models can be improved dramatically with relatively simple steps of fine-tuning on a range of standard linguistic tasks. we note that there are differences between the best performing fts for the prt and apd methods. although certain fts are shared (e.g., nli and pos), others appear more systematically in prt or apd. we interpret these inconsistencies as stemming from differences between the prt and apd methods themselves, and do not view them as negative. instead, each method enriches the baseline with different types of information and hence allows the model to capture slightly different aspects of lsc. combined with the fts, the final results can be quite different. this comple-menting view of the two lsc methods is supported by the results of @xcite , who recently reported that joining prt and apd improves lsc detection results. the most probable explanation for this is that the two lsc methods are sensitive to different aspects of change. from a theoretical point of view, our conclusions are inline with how linguists describe the phenomenon of lsc. linguistic theory distinguishes between different types of lsc, and emphasizes that changes are never "general" but pertinent to certain aspects of meaning. therefore, computationally analyzing words' meaning change using a "single ruler" as is done by current state-of-the-art lsc models, may be insufficient to describe the richness and diversity of change. we believe our findings provide an inroad for extending the capacity of lsc models and encouraging future research in this direction. this is but the first step in exploring the potential of using ft to enrich and improve contextualized models of lsc. in our future work we will corroborate these findings more rigorously: extending these to other languages, testing the generalizability of the chosen ft tasks across lsc models, and test our approach in the discovery of new cases of words that change their meaning to go beyond a small set of examples.