An Evaluation Framework for Mapping News Headlines to Event Classes in a Knowledge Graph (2023)

Abstract: Mapping ongoing news headlines to event-related classes in a rich knowledge base can be an important component in a knowledge-based event analysis and forecasting solution. In this paper, we present a methodology for creating a benchmark dataset of news headlines mapped to event classes in Wikidata, and resources for the evaluation of methods that perform the mapping. We use the dataset to study two classes of unsupervised methods for this task: 1) adaptations of classic entity linking methods, and 2) methods that treat the problem as a zero-shot text classification problem. For the first approach, we evaluate off-the-shelf entity linking systems. For the second approach, we explore a) pre-trained natural language inference (NLI) models, and b) pre-trained large generative language models. We present the results of our evaluation, lessons learned, and directions for future work. The dataset and scripts for evaluation are made publicly available.

## introduction businesses and organizations can benefit from seeking knowledge of new events that may have an impact on their business. to assist in this task, there are several media monitoring solutions with features that can provide alerts and real-time analysis for ongoing events. the majority of existing solutions are centered around entities and/or topics. for example, they identify mentions of key companies or people, group texts by topics, and analyze contents for sentiment. on the other hand, there is great value in an event-centric solution that identifies ongoing events and analyzes the characteristics of the identified events to enable event-based reasoning. in particular, such a solution would enable causal reasoning to determine the causes and consequences of ongoing events and identify potential risks and opportunities @xcite . to enable a knowledge-driven event-centric news analysis and monitoring solution, a key re-quirement is the ability to accurately map ongoing news to event-related classes in a knowledge base. one way to perform this mapping is to treat eventrelated classes as a set of categories (or topics) and classify news headlines into these categories. prior work has studied classification methods for news headlines (e.g., see @xcite ; @xcite and references therein). the majority of existing methods rely on supervised learning and therefore require a training corpus. for a generic solution that can adapt to changing event classes or one that can be tuned easily for different domains, it is not feasible to rely on the availability of training corpora large enough for accurate classification. in the absence of training data, the alternative solution is to apply unsupervised or weakly supervised classification methods that rely on little or no training data. such methods often rely on rules and pre-trained generic models. more recently, pretrained language models, and in particular large language models, have shown superior performance in such settings. as a result, we have seen a surge in the number of available models, each using different architectures, parameters, pre-training corpora, and fine-tuning strategies. choosing the right model for a given task requires an evaluation framework to measure the accuracy of the models on the end task. in this paper, we present an evaluation framework for unsupervised mapping of news headlines to event classes in a knowledge graph. to the best of our knowledge, this is the first benchmark dataset and evaluation framework for this task. in what follows, we first present the task definition and use cases we envision for the task. we then describe our methodology for creating the benchmark dataset. next, we present the results of our evaluation of a number of methods belonging to two different kinds of unsupervised techniques. we discuss key lessons learned and a number of avenues for future work. the datasets used in our major explosions at uk oil depot mass shooting takes place in new york city subway system id: q179057 label: explosion id: q21480300 label: mass shooting news headlines wikidata news event entities russia withdraws from chernobyl in ukraine label: war id: q198 experiments as well as the evaluation framework are publicly available @xcite . our target task in this paper is as follows: given a news headline and a set of event classes from a knowledge graph, find the most relevant event class to the news headline. the news headline is a short text (typically a sentence) that indicates the content of a news article by providing a concise summary of the article's contents. the knowledge graph contains event-related classes. each class comes with one or more labels, a description of the class, and possibly a class hierarchy and other attributes. figure 1 shows examples of news headlines, event classes, and their mappings. we refer to this task as news headline event mapping. note that this task is different from the event linking task defined by @xcite which takes an event mention (a phrase) and a context as input, and finds a specific wikipedia article as output. nevertheless, as described in section 4.1, such methods can be used for our task. figure 2 shows example use cases for news headline event mapping in the context of a knowledgebased news event analysis solution @xcite . in this context, news headlines from a variety of sources or a news content aggregation service (e.g., eventregistry @xcite ) are monitored in order to identify major news that could have an impact on a users' organization, on a certain region, or more generally on society. this domain of interest is defined through a knowledge graph of events that contains a rich source of knowledge about past events and event classes. such a source of knowledge can be gathered through automated knowledge extraction methods @xcite or be derived from domain-specific or general-domain knowledge sources such as wikidata @xcite . the knowledge graph provides event classes along with labels and descriptions to be used for news headline event mapping. the output of headline event mapping is then used for an analysis of the potential causes and effects of the identified event. the outcome can be used as a part of a news monitoring solution to create alerts for the identified event or its consequences so that it can assist with managing a potential risk or opportunity. it can also provide the required knowledge for an analyst looking at the implications of ongoing news for a business or organization. finally, it can be used as an input for scenario planning @xcite or event forecasting @xcite . ## benchmark dataset to the best of our knowledge, there is no benchmark dataset for the task of news headline event mapping. there are benchmarks on related tasks such as entity linking @xcite that include news headlines. however, none of these benchmarks provide ground truth event class annotations. we have, therefore, curated a new dataset designed for the news headline event mapping task using wikidata and wikinews. first, we leveraged the links to wikinews articles in wikidata to gather a collection of event-related instances. to focus on event classes, we then filtered out instances that are not subclasses of occurrence (q1190554) as well as classes with very short labels, as some non-event related entities are also linked to wikinews. finally, these articles were reviewed manually to check whether they related to news headlines and news events. this yielded 105 wikinews headlines mapped to wikidata event classes. we manually added five headlines from other sources for a final dataset of 110 mappings of headlines to wikidata event classes. the labels of all of the wikidata classes included in our benchmark are shown in figure 3 . the examples of figure 1 were taken from the dataset. there are a number of other benchmark datasets in the literature for related tasks. closest to our task is that of zero-shot sentence classification. @xcite present an excellent review of benchmarks for this task. many benchmarks for news headline classification and for zero-shot sentence classification target binary classification (e.g., for emotions or sentiment or clickbait detection), or a small number of topics. the closest to our benchmark is the yahoo! dataset @xcite , which con-sists of 10 topics. to our knowledge, there is no benchmark that targets the task of assigning news headlines to event classes in a knowledge graph or a large number of well-defined topics. ## evaluation we use our benchmark and evaluation framework to evaluate the effectiveness of a number of different kinds of methods in news headline event mapping. we first describe the approaches and implementation details for each method. we then present the results of the evaluation and a detailed discussion on key lessons learned and directions for future work. ## conclusion in this paper, we defined the task of news headline event mapping and outlined a few use cases for the task in event monitoring, analysis, and forecasting solutions. we presented an approach for creating a benchmark dataset, and used it to create the first benchmark dataset for the evaluation of news headline event mapping methods. we used the benchmark to evaluate different classes of mapping methods, including a) zero-short classification based methods, b) adaptations of classic entity linking methods, and c) methods based on large generative language models. our results provide interesting insights on the strengths and weaknesses of each of the methods. we outlined several avenues for future work, including our plan to extend the dataset, work on an ensemble method, and further experiments on llm-based methods. our dataset, as well as our evaluation script and outputs of the models, are publicly available on our github repository.