Diachronic Embeddings for People in the News (2020)

Abstract: Previous English-language diachronic change models based on word embeddings have typically used single tokens to represent entities, including names of people. This leads to issues with both ambiguity (resulting in one embedding representing several distinct and unrelated people) and unlinked references (leading to several distinct embeddings which represent the same person). In this paper, we show that using named entity recognition and heuristic name linking steps before training a diachronic embedding model leads to more accurate representations of references to people, as compared to the token-only baseline. In large news corpus of articles from The Guardian, we provide examples of several types of analysis that can be performed using these new embeddings. Further, we show that real world events and context changes can be detected using our proposed model.

## introduction diachronic embeddings are an extension to traditional word embeddings that capture changes in word representations over time. these approaches have been used for several computational social science studies focused on the analysis of language and its change over time. for example, @xcite used embeddings to study changes in gender and racial biases over decades of literary documents, and @xcite used diachronic embeddings to solve temporal word analogies, leading to insights about political and social changes. in computational studies of linguistic change, news corpora have been a popular resource because of their stylistic consistency from year-toyear and the availability of large amounts of text from each individual year or even month. in the news, changes in the usage of a word often correspond to changes in the world as well, allowing for inferences about what is happening in the world, and what journalists have been focused on from changes in the embeddings over time. previous work has also analysed the changes that representations of specific entities undergo, such as corporations like amazon and apple or even names of people such as obama or trump @xcite . however, linking the surname of a person directly to a specific individual is problematic in many cases. it does not allow us to have distinct embeddings for people with the same surname, such as bill and hillary clinton. a surname might also be a word, such as "may". there is theresa may, the person, but also the month may and the verb may, which are often treated as the same token for the purposes of creating word embeddings. issues like these can interfere with downstream analyses based on diachronic embeddings, leading to cases where multiple distinct embeddings exist for the same person, causing information about them to be potentially overlooked. at the same time, multiple people may be represented by the same embedding, resulting in noisy results without a clear way to determine the full set of people represented by the embedding, or a way to disentagle each person's influence. in this work, we tackle these problems by creating explicit, diachronic embeddings for references to individual people over time, which we embed in the same diachronic space as the remaining words. we do this by finding mentions of people in texts, then linking various surface forms of the same person together and aggregating these contexts. we use named entity recognition and use full names to alleviate the problems mentioned above. all work is done on a data set consisting of more than 2 million articles from the guardian, a british newspaper which has not previously been used in studies of diachronic embeddings. this provides an additional perspective to a number of studies which have exclusively focused on us-centric media outlets @xcite , especially the new york times @xcite @xcite . further, we provide a set of case studies, showcasing some of the main issues with the baseline approach and the kinds of analyses that can be performed with our diachronic embeddings. 1 2 related work news corpora have seen a lot of attention from researchers in diachronic embeddings to make a variety of inferences about the real world. from finding temporal analogies @xcite ) -such as "ipod" being the 2000s equivalent of a "walkman" in the 1990s -observing the change of the words "amazon" and "apple" @xcite or tracing armed conflicts @xcite . @xcite explicitly also show examples of persons in the media changing contexts as well as the changes in the association of a role to a person (president, mayor). we use their dynamicword2vec (dw2v) model to create the diachronic embeddings. compared to other models which are typically based on alignment of trained embeddings spaces, dw2v aligns embedding spaces during training. @xcite provide a good overview of the field. @xcite do event detection instead of gradual context changes, by tracking country names in the news over time and observing state changes between war and peace. they propose the aggregation of multiple words into "concept embeddings" and manage to improve the event detection score significantly this way. this is an example of a move away from strict word embeddings towards more high-level embeddings. previous analysis of people in corpora as mentioned above is strictly based on the simple mapping of a token to a person, usually the persons surname is used @xcite . while this is a workable solution, we seek to show how diachronic embeddings can be improved with more explicit linking of references to people. a similar task to linking references of people in the news for diachronic embeddings is literary character detection @xcite for modeling relationships between characters over time @xcite . however, proposed approaches such as the book-nlp pipeline @xcite are focused on very long texts with set of recurring characters with less ambiguous names. in news data, on the other hand, it is common for completely unrelated people to share surnames (or even full names), and the same person may be mentioned in a large number of separate documents. ## data we use a data set sourced from the british newspaper the guardian. the guardian provides all their content via an api called openplatform foot_0 , launched in 2009 @xcite . this data source has seen only tangential use in the scientific community @xcite @xcite and has not been used for diachronic models before. the data set contains a total of 2, 021, 947 articles spread over the years 2000 to 2019, containing 1.65 billion tokens. the documents foot_1 were retrieved from the api in march 2020. each document consists of the article body and additional meta-information such as the section in which the article was published. we used spacy foot_2 to tokenize the text, and the resulting data was a collection of token sequences, divided into yearly chunks. figure 1 shows the distribution of articles over the years, each year contains about 100, 000 articles, which is about 270 articles every day. the distribution of articles over sections is shown in figure 2 . articles are distributed very broadly over topics ranging from media over sports to politics. ## modelling people in the text while it is common for a person to be only referred to by their surname, a reference to a person in a text can take different forms, which should be linked to a single entity to allow the creation of consistent embeddings for people's names. once a set of mentions have been linked to a person, a new text corpus is built where any span of tokens that is part of a mention is replaced by a pseudo-token representing the person. ## diachronic model once persons have been identified and linked, we trained diachronic models with the dynamic-word2vec (dw2v) model @xcite . the model learns embeddings for all time slices concurrently, encoding within-time-slice similarities of tokens as well as inter-time-slice similarities of tokens to themselves at the same time. this eliminates a subsequent alignment step and also makes embeddings more stable. the experiments and analysis of the models investigate the following questions both qualitatively and quantitatively: (1) how well can the explicit person model generate traces for people, compared to a token based baseline? (2) how well do the traces model context changes in the real world? ## discussion of results in the analysis of the persons that appear in the corpus we showed the extend to which duplicate names are prevalent, showing that more than a third of people do not have a unique surname. in the examples from the embeddings we showed how these ambiguous and duplicated surnames prevent meaningful embeddings for these names in the baseline model. with the example of the uk prime minister we showed that our model identified the prime minister each year with 90% accuracy compared to 65% for the baseline model. for people with unique but potentially ambiguous surnames, our model improved embeddings as well. improvements were minor, but consistent across 15 out of 16 different experiments. for the analysis of the change tracking in the model, the case of the prime minister of the uk showed that the explicit person embeddings improved the association markedly. a systematic look at change detection in role changes for football players showed performance four times better than the random baseline, but there is still room for improvement. context changes in the model can be linked to real world events, but not all real world changes appear in the model. this may be explained by selective reporting in the source corpus, as not all events that actually happen are reported on. reporting is also selective in other ways: for a role change like an election and a new prime minister, the reporting focuses more on the new person overtaking the role than on the person being replaced. when a prime minister changes, there is a much more noticeable context change for the person getting into office than the person getting out of office. for athletes, team changes or role changes are also less easy to detect than complete out of context reporting, about for example sexual misconduct or drunk driving. overall, the detection and linking of names worked well and was a large improvement over using surnames only. it enabled us to identify certain people in the first place and disambiguate people with common surnames. the ner has a few false positives, but false positives are consistent (i.e. twitter is misclassified as a person, but in every occurrence). ## conclusion we have shown that full name identification of individuals in the news based on ner and heuristic linking is doable and provides meaningful insights linkable to real world developments. this approach is a simple yet effective improvement over a token based baseline. we quantified some common problems with the baseline approach, such as duplicate and ambiguous surnames. we provided some initial analysis of event detection both based on samples and a small quantitative experiment, showing promising results, and we provided an analysis of a previously understudied, british news corpus. future work on the model can incorporate improvements to ner models as well as expanding the detection of references to non-name references such as role references, co-references. linking can be improved with a more sophisticated treatment of name variants and titles. these steps have been shown to improve character detection in literary texts @xcite . lastly, gaps in reporting could be filled by relying on more than a single newspaper as a source of text.