Machine Reading of Historical Events (2020)

Abstract: Machine reading is an ambitious goal in NLP that subsumes a wide range of text understanding capabilities. Within this broad framework, we address the task of machine reading the time of historical events, compile datasets for the task, and develop a model for tackling it. Given a brief textual description of an event, we show that good performance can be achieved by extracting relevant sentences from Wikipedia, and applying a combination of task-specific and general-purpose feature embeddings for the classification. Furthermore, we establish a link between the historical event ordering task and the event focus time task from the information retrieval literature, showing they also provide a challenging test case for machine reading algorithms.

## introduction machine reading concerns the extraction of entities and relations from text and the ability to use them meaningfully, for instance by answering questions based on them, inferring other relations from them, or using them to compile knowledge bases. such an inclusive task definition necessarily builds on a wide range of nlp capabilities, from syntactic and semantic analysis, to the use of world knowledge and common sense. the inclusive nature of the task supports the development of general-purpose methods, but also results in low performance in absolute terms, difficulty in defining widely agreed-upon evaluation protocols, and difficulties identifying the sources of prediction errors @xcite @xcite . this paper addresses a sub-task of machine reading, namely the task of estimating when a historical concretely, given a short text description of a historical event, and an external data source (henceforth contextual information or ci), the task is to predict the year in which the event happened. the external source in our case is wikipedia. for example, given the event description "the government of turkey expels patriarch constantine vi from istanbul," the task is to infer the year it took place @xcite . we select wikipedia as a source for contextual information, due to its broad coverage, and the wide interest it receives in the nlp community. indeed, wikipedia has often featured as a semi-structured knowledge base, e.g., as a source of concept grounding @xcite and indirect supervision @xcite . we hypothesize that aside from time expressions, the ci words themselves give an approximate time in which an event happened. for example, the pres-ence of the word "spacecraft" in the ci probably indicates an event that occurred after 1900, while the presence of the word "sword" most likely indicates an event that occurred before 1900. the task is therefore different from tasks addressing the extraction and normalization of time expressions, or from related tasks pursued in the context of information retrieval (see §8). our results support this hypothesis, and demonstrate that even when time expressions are not present in the data, it is still possible to predict the approximate year in which an event happened. we compile two datasets for the task, based on the websites "wikipedia on this day" webpages (wotd) and "on this day" (otd). we consider wotd as an in-domain setting, given that it is taken from wikipedia as well (albeit from an entirely disjoint part of wikipedia). the otd setting was selected to be maximally challenging for leveraging external data sources, since (1) event descriptions are taken from a different website, and may be formulated very differently from wikipedia; (2) it is an order of magnitude larger, and so the classifier has plenty of data to train on, even without relying on external data sources. our results show that on wotd, good performance can be obtained by detecting relevant sentences from wikipedia and extracting year mentions in them, but that substantially better performance can be reached when additionally encoding the entire sentences, using neural machinery. in otd, ci yields more modest improvements. results in absolute terms are high: the best models obtain a mean kendall's τ correlation with the correct event ordering of 0.77 (wotd) and 0.71 (otd). ## task definition and motivation the historical event ordering (heo) task is defined as follows. given a set of brief event descriptions and some textual resource, the task is either to predict the year in which each event occurred, or to find a ranking of events such that they are ordered by date of occurrence. the first variant is stronger than the second, as it implies a ranking. our evaluation uses both rank correlation (kendall's τ ) and measures of the distance between the year the event took place and the predicted year. see section 5.2 for details. differences from question answering. while traditional question answering tasks require the answer to be in the text (e.g., @xcite , the heo task is based on estimating the time of occurrence of an event. this estimation is based solely on lexical cues, and does not require an explicit answer in any text. this is a major advantage of heo models, as explicit answers are not always present in the text for two reasons: (i) we would need a massive amount of text for good coverage of historical events, which may be unfeasible to use in real-world applications; and (ii) new events are constantly occurring, and existing machine reading comprehension models will invariably fail on those (e.g., "when was donald trump elected president?" will not be covered in old data, but could be inferred to have happened recently based on recognizing the named entity "donald trump"). as answers are not guaranteed to be in the text, the heo task is somewhat more challenging than traditional question answering tasks. the task's challenge is also evidenced in that it requires temporal commonsense reasoning and in being challenging for humans (see §6). real-world applications. as previously mentioned, heo models do not assume the presence of the answer in the source text, and can thus be used for temporal question answering when the answers are not present in it. by leveraging the lexical information that exists only in the question itself, these models can serve as a fallback for such cases. other possible applications are dating of historical documents based solely on the documents' text, improving search engines that leverage the time of queries @xcite , as well as making inferences that involve rough temporal placement of the statement (e.g., inferences involving refrigerators are unlikely to be relevant before the 20th century). ## data collection this work introduces two datasets: wotd and otd. despite the similarity in their names, we are not aware of any influence or other relation between them. using both datasets thus makes our experimental analysis less prone to be biased by dataset-specific artifacts. wikipedia on this day (wotd) was scraped from wikipedia's on this day webpages. 2 the dataset contains 6,809 entries. some example entries are presented in on this day (otd) is a scrape of the on this day -today in history, film, music and sport @xcite . 4 on this day has a dedicated team that adds, verifies content, and responds to corrections from the public. 5 the dataset contains 75,135 entries consisting of a sentence describing the event and the event's date. we removed 96 events from the original dataset, which happened bce (before common era), and also removed events that had not happened yet. the earliest event in the dataset occurred on year 1 ce (common era), and the latest occurred in 2018 ce. the median label is 1960.0, while the mean label is 1913.8, so the distribution of labels is not uniform: there are more events occurring in recent times. the standard deviation for the labels is 172.3 years. examples of entries from otd are presented in figure 1 shows the distribution of event years in otd and wotd. both datasets have significantly more recent events from the last few centuries. we use a random 80/10/10 split of each dataset to form the training, validation and test sets. ## algorithmic approaches we propose two models: a bag of embeddings model (boe) and a recurrent neural network model (lstm). both take a training example and output a timestamp, in our case the year of the event. we explore two supervised settings: a classification setting, where each possible year corresponds to a different class, and a regression setting, where the labels are the numerical value of the timestamp. as baselines, we define two models: one predicts the mean year of the training set (mean), and one predicts the median year present in the extracted ci, falling back to the other baseline if no years are found (ciyear). ## experimental setup in this section we describe our experimental setup and the evaluation metrics we use. ## results our results show that the wikipedia enrichment is an essential component of the protocol. for the wotd dataset, all models exhibit a statistically significant improvement in ordering when adding ci, with the smallest improver being the lstm classification model, with a +0.053 change in τ , and the largest improver being the boe regression model, with a change +0.098 in τ . for the otd dataset, the lstm model showed a modest but statistically significant improvement when adding ci. the boe model presents a minor decrease in performance; however, we obtain a statistically significant improvement of +0.027 in τ by restricting the ci to only include year mentions. 11 as the otd and the extracted ci are from different domains, the words of the contextual information most probably add too much noise for the boe model to handle, which is why a performance improvement is observed when only including years, which are not domain-specific. this indicates that leveraging ci is important, even in this more challenging scenario, where the training data is large and the ci is from another domain, but also suggests that additional improvements, such as using domain adaptation techniques (ziser and reichart, 2017) for bridging the domain difference, are required to obtain better performance. one difference between the regression and classification settings is that the latter has higher exact match metrics than the former. this reflects the nature of the two architectures: when using l1/l2 regression, the loss is proportional to the difference in the prediction, whereas in classification what matters is the probability assigned to the exact year. on the whole, the lstm model produces better predictions than the boe model, according to most measures. this is perhaps unsurprising, as it is able to capture word context when analyzing the inputs, leading to more effective reasoning. ablation study. we observe that for both models, the impact of removing the event text and using only the extracted contextual information leads to a τ change of -0.043 for boe and -0.071 for lstm. this shows that the heuristics we propose for extracting ci are effective at retrieving relevant information. study b was conducted on all datapoints from the wotd dataset. we report the impact of removing tokens denoting years, dates and numbers from both the ci and the event text. we remove years using the method described in §4.1. we remove dates by removing any tokens within a date entity. we remove numbers using the like num property of the spacy tokenizer, which includes different forms that may be considered numerical (e.g. "1" and "one"). clearly, the removal of dates subsumes the removal of years, and we expect the removal of numbers to remove at least part of all dates, including years, alongside other date-unrelated numbers. the removal of these features has a very similar impact. this is particularly the case for the lstm model, where the change in τ was -0.041, -0.042 and -0.051 when removing years, dates and numbers, respectively. boe presents similar differences in performance when removing those features, with a change in τ of -0.045/-0.031/-0.054 when removing years/dates/numbers. these results support our hypothesis that substantial information about the time of an event is encoded in the vocabulary used, and not only in the time expressions. human performance we compare our results to human performance on this task. three participants were given 100 randomly selected events from the wotd dataset and were asked to predict years of occurrences, without using any contextual information. all participants consider themselves as having good knowledge of history, but are not history experts. on average, their error was 52.3 years. the participant who had the best results had a mean error of 34.6 years, which is only 3.8 years less than our best result on the wotd dataset. ## qualitative analysis in order to demonstrate the challenges put forth by the addressed task, we examine some events from the otd development dataset on which our best performing models, lstm regressor and boe classifier, got significant prediction errors. we observe that some events contain words that are usually associated with a different period in time than the year the event occurred in. for example, "portuguese expel jesuits" occurred in 1911, but most jesuits-related events in our training data occurred in the 16th century. one of these events which is particularly similar to the above is "english parliament expels jesuits", which is dated to 1584. probably for these reasons the lstm and boe had similar outputs for this event -1559 and 1581, respectively. another example for such an event is "order of merit instituted by king edward vii", which occurred in 1902, but the word "king" normally appears in events dated to earlier centuries. the lstm model output for this event is 1527, and boe model output is 1639. both events had no ci extracted for them, therefore the models had to rely on words in the event description only. an example for which relevant ci was extracted but the models still erred substantially is the event "all female jury hears case of judith catchpole accused of killing her child (acquit her) in patuxent county, maryland". this event is dated to 1656, but the boe model prediction for the event is 1957, and the lstm model prediction, 1873, is only slightly better. the contextual information extracted for this event was "upon her arrival she was accused of several crimes, resulting in a trial on september 22, 1656 in the general provincial court in patuxent county, maryland". the exact date of occurrence does appear in the extracted data, and still both models have a substantial prediction error. this is probably due to the fact that our training data contains many "court" and "jury" related events, where most events containing "court" are relatively recent (19th century and later), and almost all "jury" related events are dated after 1900. in some cases, the extracted ci can mislead our models. for the event "scotland and france form an alliance, the beginnings of the auld alliance, against england" that occurred in 1295, lstm predicted the year 1659. five sentences were extracted for this event, which contained the years 1603 and 1707. another example is "over 250 years after their deaths, william penn and his wife hannah callowhill penn are made honorary citizens of the united states" occurred in 1984. the ci extracted includes the exact true date of the event, but also includes information regarding the penns' lives, and contains years ranging between 1680 to 1726. this is probably the cause of error for the boe model, which predicts the year 1721, whereas the lstm model may have been able to better filter the correct ci, and predict the year 1921. errors can also arise from terms that are ambiguous between time periods. "queen elizabeth" is such a term: it can indicate an event from the 16th century, but also an event from the 20th/21st centuries. indeed, we notice confusion of the boe model on events related to queen elizabeth. for example, "francis drake knighted by queen elizabeth i aboard golden hind at deptford" occurred in 1581, but the boe model predicts the year 2013 -even though the true target year appears in the extracted ci for the event: "i visited the royal dockyard on 4 april 1581 to knight the adventurer francis drake." similarly, the event "ted hughes is appointed british poet laureate by queen elizabeth ii" occurred in 1984, but the boe model predicts the year 1579, which corresponds to queen elizabeth i. we note that for those two events the lstm model gave better predictions (1566 for the first event and 1981 for the second), which may be related to the inherent difficulty of boe to address multi-word expressions like "queen elizabeth i". ## related work work on event ordering can largely be categorized into event ordering in context, which aims to order event instances within a given text or discourse and is tackled as part of the tempeval shared tasks @xcite , and lexical event order-ing @xcite , which attempts to order event types by their prototypical temporal order. somewhat in between these lines of work is crossdocument event ordering @xcite , which orders events that are mentioned across different documents. however, this task does not rely on machine reading external textual resources as we do here, and does not focus on historical events that by their nature are described in a variety of (often incompatible) ways. a related line of work to ours was pursued in the context of information retrieval (ir). @xcite tackled the task of estimating what the "focus time" of a given document is. focus time is defined as the time to which the main event addressed by the document refers to. they do so by computing the association of words and time expressions, based on their co-occurrence, using a bag-of-words method. @xcite address the task of focus time prediction for short event descriptions, which resembles the task at hand. they do so by using cosine similarity to rank a set of candidate years for each event, all of which are computed using word embeddings. in a similar vein, @xcite find the focus time of short event descriptions by relying on year mention statistics in related wikipedia articles and dbpedia entries. while these two works are related to our task in spirit, our work is not an instance of the event focus time (eft) task. in fact, we believe the eft task can be seen as a special case of the heo task. this is evidenced by the approach of eft systems, which exhibits traditional ir design and techniques, such as producing a ranking of candidate predictions for each document, and is evaluated using ranking-specific metrics that forbid system designs such as predicting years using regression. as heo subsumes eft, we attempted to evaluate the performance of eft systems in the heo task, but have been unable to obtain code for either of the systems. we have also been unable to reimplement the systems: @xcite leaves implementation details unspecified, and @xcite utilizes a proprietary system. another related line of work seeks to create timelines of temporal events by predicting their starting and ending points. @xcite address the problem of ensuring semantically consistent timelines by finding patterns in the ordering of endpoints of different event types, which adds a common sense reasoning component to the system. @xcite construct a relative timeline of events directly, which allows them to circumvent typical pitfalls of pair-wise classifiers, such as computationally intractable inference and constructing globally inconsistent orderings (with cycles). our work takes a similar approach but instead is able to construct an absolute timeline for the restricted domain of historical events. within the domain of temporal text understanding, the extraction and normalization of temporal expression may inform the task at hand. for example, @xcite defined the task of tagging temporal expressions, which are named events or facts with temporal scope, such as "second term of angela merkel". they used a rule-based system to detect such expressions in free-text and map these expressions to a knowledge base (kb) containing time scopes of temporal events and facts. this approach requires the existence of kb records containing time scopes for the events. ## conclusion in this paper we argued that the task of predicting the time of historical events strikes a balance between being a focused task, with transparent evaluation and interpretable results, and presenting challenges that are not simple to overcome using standard nlp models. we outlined a procedure to extract the ci related to an event and compared two approaches for the task, using bag of embeddings and an lstm, showing that the latter achieves the best performance. future work will explore the use of domain adaptation techniques to enhance performance where the domains of the ci and event text differ substantially. matthew e. peters, mark neumann, mohit iyyer, matt gardner, christopher clark, kenton lee, and luke zettlemoyer. 2018. deep contextualized word representations. in proc. of naacl. pranav rajpurkar, jian zhang, konstantin lopyrev, and percy liang. 2016. squad: 100,000+ questions for machine comprehension of text. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383-2392. gabriel stanovsky and ido dagan. 2016. creating a large benchmark for open information extraction. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 2300-2305. naushad uzzaman, hector llorens, leon derczynski, james allen, marc verhagen, and james pustejovsky. 2013. semeval-2013 task 1: tempeval-3: evaluating time expressions, events, and temporal relations. in *sem-semeval '13, pages 1-9. yftah ziser and roi reichart. 2017. neural structural correspondence learning for domain adaptation. in proceedings of the 21st conference on computational natural language learning (conll 2017), pages 400-410.