Rethinking the Event Coding Pipeline with Prompt Entailment (2023)

Abstract: For monitoring crises, political events are extracted from the news. The large amount of unstructured full-text event descriptions makes a case-by-case analysis unmanageable, particularly for low-resource humanitarian aid organizations. This creates a demand to classify events into event types, a task referred to as event coding. Typically, domain experts craft an event type ontology, annotators label a large dataset and technical experts develop a supervised coding system. In this work, we propose PR-ENT, a new event coding approach that is more flexible and resource-efficient, while maintaining competitive accuracy: first, we extend an event description such as “Military injured two civilians” by a template, e.g. “People were [Z]” and prompt a pre-trained (cloze) language model to fill the slot Z. Second, we select suitable answer candidates Zstar = “injured”, “hurt”... by treating the event description as premise and the filled templates as hypothesis in a textual entailment task. In a final step, the selected answer candidate can be mapped to its corresponding event type. This allows domain experts to draft the codebook directly as labeled prompts and interpretable answer candidates. This human-in-the-loop process is guided by our codebook design tool. We show that our approach is robust through several checks: perturbing the event description and prompt template, restricting the vocabulary and removing contextual information.

## introduction decision-makers in politics and humanitarian aid report a growing demand for comprehensive and structured overviews of socio-political events @xcite . for this purpose, news papers are automatically screened for event mentions, a task referred to as event detection and ## event data and types we consider a subset of the armed conflict location and event data (acled) @xcite dataset. it is widely-used and has large coverage of political violence and protest events around the world. each event is human annotated with a short description, its event type and additional details such as the number of fatalities and actor and targets. the event types are based on acled's own event ontology which distinguishes 6 higher-level and 25 lower-level event types. some event types are easily separable (e.g. protests vs battles), while others are harder to distinguish semantically (e.g. protests vs riots) (see fig. 9 in the appendix). we sample 4000 acled events (3000 for training, 1000 for testing) in the african region while maintaining the event type distribution of the full dataset (see fig. 9 ). we remove empty event descriptions and annotator notes (e.g. "[size: no report]"). in fig. 8 in the appendix, we present statistics of the test set, showing different aspects of linguistic complexity. in §4.2, we consider the global terrorism dataset (gtd) @xcite to study the effect of domain shift. ## entailment-based prompt selection our approach, pr-ent, represents a real-world use case of prompting and textual entailment to code event descriptions @xmath0. 1b . ## evaluation: event classification we compare pr-ent against the conventional event coding pipeline in an evaluation along three dimensions: accuracy, flexibility and efficiency. ## human-computer codebook design to make use of pr-ent, domain experts need to design a codebook (i.e. a mapping), between event types and entailed answer candidates. creating this mapping is non-trivial as there exists a trade-off between interpretability and accuracy. in essence, a codebook is interpretable when the answer candidates are representative of the corresponding event type. a bad codebook contains a large number of non-readable entailed answer candidates. a codebook is accurate when a few answer candidates are sufficient to allow for a clear differentiation of the event types. to that end, we propose an interactive codebook design tool foot_1 that helps designing templates and answer candidates by presenting accuracy metrics. the assessment of interpretability is left to the human domain experts. codebook design. our codebook is a mapping between event types and entailed answer candidates. for example, an event can be classified as "kidnapping" if any of the following templates is the dashed line corresponds to all kidnapping events coded by acled annotators. the blue line corresponds to all kidnapping events coded by pr-ent. we find that the positions of the time series spikes between pr-ent and acled's coding align well. however, the spikes in the pr-ent time series are higher indicating that pr-ent detects more events. this may be due to more granular event coding or the advantage of not being limited to only one event type per event description. as a second step, domain experts select some of the entailed answer candidates provided by the model. if no entailed answer candidate is informative to classify the event, it is possible to group multiple entailed answer candidates with an and condition. for example, "riot" event types can be coded with the two following templates: "this event involves @xcite ." and "this event involves @xcite .". the tool also offers the possibility of excluding certain answer candidates. ## discussion is this few-shot, unsupervised tagging? while we have evaluated accuracy, efficiency and flexibility, it is up for discussion and definition whether our approach should be considered few-shot, unsupervised or tagging-based. in some cases, the language model copies tokens verbatim from the input, which could be seen as a form of "event tagging". in other cases, the answer candidates are abstract tokens outperforming purely tagging-based approaches. in cases where the answer candidates map directly to an event type without an additional shallow classifier §4.2, our approach may be considered unsupervised and zero-shot. on the contrary, the template is designed in an iterative trial and error fashion. thus, it is tuned to observed data instances which arguably violates the zero-shot setting and should be framed few-shot instead. entailment-only approach. the presented approach pr-ent relies on textual entailment to select entailed answer candidates from prompts as motivated in §3.2. however, textual entailment could have been considered for classification by itself @xcite . in this setting: a predefined set of hypotheses is created for each event type and is tested against each event description. however, this reduces flexibility as we need to define a broad set of hypotheses in advance. our prompting-based approach relies on large language models which do not require labeled training data for training. as a consequence, they are more frequently updated and trained on larger amounts of data. ## related work similar to our prompting-based approach, existing work evaluates off-the-shelf qa @xcite and nli @xcite models for event coding. the prompting approach shares similarities with @xcite , who build a semantic parser to map natural text to canonical utterances. their training set is constructed by prompting a language model in a human-in-theloop fashion. @xcite uses nli to extract relationship between two given entities based on a predefined hypothesis template. @xcite present an approach to identify words that can serve as high-accuracy labels for text classification. however, they are not focusing on interpretability and a particular application domain such as political event coding. there also exist methods for automating prompt generation and selective incorporation of examples in the prompt @xcite . existing work in promptbased classification focuses on sentiment, topic or intent @xcite @xcite . within the field of event coding, we distinguish work on event detection, event type ontologies, and automated event coding tools. our work falls into the latter two. the world event/interaction survey (weis) project @xcite was pioneering in event data collection and event ontology design. the weis successor cameo @xcite is one of the most popular event ontologies until today and used by icews @xcite and navco @xcite among others. vra-reader @xcite is among the first to automatize event coding based on matching string patterns. its successors bbn accent @xcite , tabari and petrarch2 @xcite rely on lambda calculus-based semantic parsing. recent event coding systems rely on supervised machine learning @xcite @xcite , word embedding- @xcite and transformer-based models @xcite @xcite . ## conclusion we proposed a method to select answer candidates from prompts using textual entailment. this combined usage of state-of-the-art tools is motivated by a real-world use case that benefits humanitarian aid efforts with scarce resources.