Political Event Coding as Text-to-Text Sequence Generation (2022)

Abstract: We report on the current status of an effort to produce political event data from unstructured text via a Transformer language model. Compelled by the current lack of publicly available and up-to-date event coding software, we seek to train a model that can produce structured political event records at the sentence level. Our approach differs from previous efforts in that we conceptualize this task as one of text-to-text sequence generation. We motivate this choice by outlining desirable properties of text generation models for the needs of event coding. To overcome the lack of sufficient training data, we also describe a method for generating synthetic text and event record pairs that we use to fit our model.

## introduction political event records that are automatically derived from text are an important source of structured data for researchers in social science. existing approaches to generating event data often rely on dictionary methods, which are brittle and go obsolete, or classifiers trained on hand-labeled text that required large amounts of expensive data. this paper introduces a new proof of concept model for generating structured event data from news text that does not require dictionaries or hand-labeled document. we generate synthetic news stories using a novel combination of rules based generation and a paraphrasing model and train a text-to-text transformer to produce event records from text. when evaluated on synthetic test data, the model correctly identifies high-level event types 83% of the time and reaches accuracies of 63% and 56.9% on the source and target actors, respectively. the article concludes with a brief real-world evaluation and a discussion of the model's limitations. ## results this section provides descriptions of our results in two experimental settings: an out-of-sample test set evaluation using data generated via the same process as the training data and an out-of-distribution case study using a small sample of real world news text. ## conclusion text-to-text is a flexible modeling task that is amenable to complicated output data types. using multiple classification heads is an alternative method for event coding text via large language model, but it offers less flexibility for future improvements. for example, a text-to-text model can be trained to generate an arbitrary number of events from a single input text. @xcite a more traditional classification-based approach is suitable for coding only up to a predefined number of events. we also appreciate that the open-ended nature of text output means that we do not need to generate all possible actor combinations prior to training as we would in a multiclass classification setup. the text-to-text model can simply append actor codes to one another as necessary, even if it has not previously seen a sample with the particular given combination. we leave a more formal evaluation of our methodology and model to a follow-up paper in which we plan to employ expert human annotators to generate comparable event data against which @xcite we have preliminary work in which we generate up to four events per input paragraph.