WIKITIDE : A W ikipedia-Based Timestamped Definition Pairs Dataset (2023)

Abstract: A fundamental challenge in the current NLP context, dominated by language models, comes from the inflexibility of current architectures to “learn” new information. While model-centric solutions like continual learning or parameter-efficient fine-tuning are available, the question still remains of how to reliably identify changes in language or in the world. In this paper, we propose WikiTiDe, a dataset derived from pairs of timestamped definitions extracted from Wikipedia. We argue that such resources can be helpful for accelerating diachronic NLP, specifically, for training models able to scan knowledge resources for core updates concerning a concept, an event, or a named entity. Our proposed end-to-end method is fully automatic and leverages a bootstrapping algorithm for gradually creating a high-quality dataset. Our results suggest that bootstrapping the seed version of WikiTiDe leads to better-fine-tuned models. We also leverage fine-tuned models in a number of downstream tasks, showing promising results with respect to competitive baselines.

## introduction handling new information is one of the most critical (and vastly unresolved) challenges in the current nlp landscape, mostly because language models (lms) such as bert @xcite , t5 @xcite , gpt-3 @xcite or palm @xcite ) can only learn from information they have seen during pretraining. this is an important limitation when it comes to dealing with updates in the world and language changes alike, since these updates, if not dealt with properly in an lm-centric system, can cause temporal misalignment @xcite @xcite , which is especially harming in knowledge-intensive tasks, such as closed-book qa. unsuprisingly, thus, there is a significant body of work concerned with, for instance, updating language models by pretraining them on in-domain data @xcite , editing specific facts @xcite @xcite , continual learning @xcite @xcite @xcite , pre-training with an objective specifically designed to handle infusion of newly coined terms @xcite , or directly modifying the attention mechanism to account for temporality @xcite . all these, in addition to the extensive body of work on diachronic and dynamic (contextualized and static) word embeddings @xcite @xcite @xcite . regardless of the method, however, a critical component of time-aware nlp is to have access to dynamically changing facts about language and the world so that lms are exposed to them. as @xcite argues, collaborative resources such as wikipedia or wikidata can satisfy this desideratum, since they provide a dynamically updated 2 life-long resource. given this, with wikitide we put forward a benchmark comprised of definition pairs annotated in terms of whether they are the same or not, and if not, if this difference can be attributed to a fundamental change in that term, event or entity (as opposed to, for instance, semantic variations such as introduction of a paraphrase or stylistic nuances). we construct wikitide in a weakly supervised manner via bootstrapping, and evaluate a number of lm-based baselines on the wikitide definitions label p def f irst "all or nothing" is a song by german dance-pop group milli vanilli. 0 p def second "all or nothing" is a song by german dance-pop group milli vanilli. p def f irst "along the navajo trail" is a country/pop song, written by dick charles (pseudonym for richard charles krieg), larry markes, and edgar de lange in 1945. 1 p def second "along the navajo trail" is a country/pop song, written by dick charles (pseudonym for richard charles krieg), larry markes, and eddie de-lange in 1945. alan sheffield ball @xcite is an american football cornerback for the jacksonville jaguars of the national football league. 2 p def second alan sheffield ball @xcite is a former american football cornerback in the national football league for the dallas cowboys, houston texans, jacksonville jaguars, and chicago bears. ## related work this paper can be broadly positioned within two areas, namely lexicograhpic definitions (understood as a lexicographic resource but also as a high quality source of information for augmenting lms), and diachronic nlp. we therefore make a clear distinction between them in the review of relevant works. definitions definitions have traditionally played a crucial role in nlp and computational lexicography. as the building blocks of dictionaries and encyclopedias, they are used when the meaning of a word is sought @xcite , and thus the task of automatically constructing glossaries and terminologies is a well established task in @xcite @xcite @xcite . however, definitions have also been leveraged to improve the quality of nlp systems. for instance, delli bovi et al. (2015) and espinosa-anke et al. (2016) harnessed definitions to build knowledge bases by extracting semantic relations from them; joshi et al. (2020) used definitions to provide additional context to lms in reading comprehension tasks; yu et al. (2021) pre-trained bert on tasks that exploit definitions, specifically seeking to improve contextual representations of rare terms; and xu et al. (2022) used definitions as the backbone of prompt-based taxonomy learning. in a parallel strand of work, others have explored definition modeling systems (i.e., given a term and potentially some context, generate a definition) @xcite @xcite @xcite , and these systems have been applied in tasks such as controlled definition modeling, e.g., jargon or varying technical complexity @xcite , as well as lexical semantics tasks like word sense disambiguation and wordin-context classification (pilehvar and camacho-collados, 2019). diachronic nlp while there is agreement in that continual learning helps to mitigate the fundamental issues of temporal misalignment @xcite and catastrophic forgetting @xcite , the availability of benchmarks for retrieving new facts and evaluating lms on their capacity to account for them is not overwhelming. social media seems to be a particularly well suited domain for exploring temporal generalization, given its naturally fast-paced nature, and so we find a number of twitter-specific benchmarks @xcite . moreover, algorithm 1 collect definition pairs 1: let p be the set of wikipedia pages 2: let d be the list of definition pairs 3: let n be the desired number of definition pairs (@xmath0, 000) foot_3 : let srp(p, tl) be a function for selecting a random page given a specific timeline tl 5: @xmath1: while |d| < n do 7: find a random @xmath2: tl y ← sortyearsascending(tl y ) 9: m ← findmedian(t) 10: p f irst ← srp(p, tl @xmath3: p second ← srp(p, tl @xmath4: p def f irst ← getdefinition(p f irst ) 13: in this context, we argue that wikipedia is indeed a valuable and underutilized resource for training and evaluating lms on their language and knowledge update capabilities. while, as jang et al. (2022) points out, not all changes in wikipedia or wikidata correspond to an actual change in the real world, we aim to alleviate this limitation by focusing on changes in definitions alone. in this way, we drastically reduce the chances of falsely confusing one superfluous change in a wikipedia entry with a change that results in a necessary update of our understanding of a concept or entity. in what follows, we discuss how we create our seed for the wikitide dataset, the algorithm for growing it, and then report on several experimental evaluation results. ## wikitide in this section, we discuss, first, the process of retrieving candidate definition pairs for annotation. then, we provide details about the annotation process, and finally, present examples and summary statistics, aimed to shed light on the properties of wikitide. the process of creating the required definition pairs of wikitide is shown in algorithm 1. in a nutshell, we start from the set p of wikipedia pages, and construct, by sampling two sufficiently distant definitions (that is, the first sentence of a wikipedia article @xmath5, a dataset d which contains 10,000 unannotated definition pairs. after this, we randomly select 30% from d for annotation, which we perform combining the annotations of 4 instances of gpt-3 @xcite foot_2 . the main motivation for "replacing" manual annotation with a lm is twofold. first, we posit that we can leverage the knowledge embedded in chatgpt's parameters about well known entities, concepts and events (well known because they have a corresponding wikipedia page). second, recent work has shown that leveraging chatgpt can outperform other annotation frameworks, for example amazon mechanical turk @xcite . the four rounds of annotations we perform differ in the instruction, as the hyperparameters remain fixed (specifically @xmath6. the instruction combines a prompt and a few examples (potentially -but not always -covering all possible labels). the specific variations involve paraphrasing some of the instructions or definitions of labels, or selecting different examples 4 . as for the labels, we define our task as a 3-label classification problem, and hence the 3 different labels (and how they are described to chatgpt) can be broadly defined as follows: the final labels are selected as follows: we only select instances labeled as class 2 if all instances of chatgpt label it as such, thus ensuring the tightest possible agreement for this label, which is both the most interesting and infrequent in the dataset. then, for the rest, we resort to the label assigned by the majority among three chatgpt annotators, and only in case of draw, we incorporate a fourth one, which acts as referee. at the end of this process, we annotate 3,000 instances out of the 10,000 initial set, with a fleiss-kappa agreement score of (fleiss, 1971) of 24.84, which according to the literature, falls within the fair agreement range. ## bootstrapping wikitide with t s being the chatgpt-annotated seed dataset in wikitide (with a label set @xmath7, 1, 2}), let ds be the remaining unannotated 7,000 instances, and @xmath8. we seek to iteratively bootstrap a development set with "high confident" predictions, starting from a seed classifier trained, in a first iteration, only on t s. we argue that this approach, which can be traced back to applications in word sense disambiguation and definition extraction @xcite , can be effectively applied to our use case as each newly bootstrapped definition pair will be reliable indicatives of the source training set, which can contribute to increase recall as the model will have seen more positive examples. as summarized in algorithm 2, the bootstrapping process requires at a minimum an annotated training set t s and an unannotated test set ds, and optionally held-out test set hs to monitor performance. at the first iteration, we set |t s| = 2160; |ds| = 7, 000; and |hs| = 840. we then fire the bootstrapping process, in which, first, a model is trained and applied on ds, then we extract the k most confident predictions for each label, append them to t s, and remove them from ds. every time we exhaust all labels in l, we evaluate a new instance of the model on hs. in terms of classifier, we select a wide range of models to evaluate, all of them based on the transformers architecture @xcite , namely bert @xcite , roberta @xcite for @xmath9: ## case study: wic-tsv the wic-tsv (word in context-target sense verification) task @xcite is a "shootoff" from the original wic task @xcite . it proposes a binary classification problem, where the input is a pair of sentences: the first one, a sentence with a target word in context, and the second one, a definition of that target word. this is a suitable test bet for a model fine-tuned on wikitide, since this is a dataset which essentially measures definition similarity. however, since wikitide is a multilabel dataset, we combine labels 1 and 2 as label 0 in wic-tsv and assume equivalence between the notion of "change" in wikitide and polysemy in wic-tsv. for our model to work, both input sentences must be definitions, however, this is not always the case in wic-tsv. to work around this limitation, we replace the non-definition sentences in wic-tsv with a definition generated using chatgpt @xcite . both sets of results (directly applying our model to wic-tsv as well as replacing one of its sentences with a chatgpt-generated definition) are reported, for train, test and development sets foot_5 (which is possible as we cast this problem as an unsupervised classification task), in the results of our experiment display the ability of the models before and after bootstrapping on all three sets (train,deveopment and test). the bootstrapped approach considerably increases the macro-f1 performance of the models with respect to wic-tsv's task 1 unsupervised setting baselines @xcite . the results also suggest that while bert shines on a few occasions, the roberta family of models show the highest performance, with roberta-large bootstrapped being the best with f1 score of 0.53. the vanilla versions of the models perform within a range between 0.30 to 0.34. the bootstrapped versions outperform their non-bootstrapped counterparts in all three datasets, with respect to f1 score. we also observe that the difference in f1 scores between before and after bootstrapped versions can go as high as 17 points, which signify that the models learns better during the bootstrapping process. finally, we also find that the larger models with more parameters outperform their distilled counterparts in most of the dataset versions. ## conclusion we propose a dataset and methodology to design a classifier for detecting temporal changes in temporal definition pairs. we use weak supervision technique by boostrapping the model an unlabelled dataset in output controlled setting. we also see that bootstrapping a model improves the accuracy of the model as well as makes the model more robust. however, the process requires more time to bootstrap the model and the success of the process depends on the initial training. although the process has its own limitations, we conclude that the idea of using a classifier to detect information changes in with respect to temporality and training it with boostrapping can result in easement of defining which information is relevant to update a model's knowledge base and can help to miti-gate the issues that a language model suffers due to temporal misalignment.