Rule Based Event Extraction for Artificial Social Intelligence (2022)

Abstract: Natural language (as opposed to structured communication modes such as Morse code) is by far the most common mode of communication between humans, and can thus provide significant insight into both individual mental states and interpersonal dynamics. As part of DARPA’s Artificial Social Intelligence for Successful Teams (ASIST) program, we are developing an AI agent team member that constructs and maintains models of their human teammates and provides appropriate task-relevant advice to improve team processes and mission performance. One of the key components of this agent is a module that uses a rule-based approach to extract task-relevant events from natural language utterances in real time, and publish them for consumption by downstream components. In this case study, we evaluate the performance of our rule-based event extraction system on a recently conducted ASIST experiment consisting of a simulated urban search and rescue mission in Minecraft. We compare the performance of our approach with that of a zero-shot neural classifier, and find that our approach outperforms the classifier for all event types, even when the classifier is used in an oracle setting where it knows how many events should be extracted from each utterance.

## introduction humans communicate with each other using both explicit (e.g., written and spoken natural language) and implicit (e.g., tone of voice, body language) modalities. while we posit that an artificially intelligent (ai) agent needs to handle both of these modalities to serve as an effective teammate on a hybrid human-machine team, in this paper, we focus on the former. to that end, here we present a case study describing our approach for extracting events relevant to team coordination (e.g., instructions, requests, knowledge-sharing statements about the locations of people and objects, etc.) in real-time from natural language dialog. this was carried out in the context of darpa's artificial social intelligence for successful teams (asist) program, foot_0 a 4.5 year program aimed at developing technologies for imbuing artificial agents with social intelligence, i.e., the ability to construct and maintain models of their human teammates in order to provide more effective assistance. the program is structured around five large-scale experiments. one of the primary goals of these experiments is to evaluate the ai agents developed in the program on their ability to successfully predict human behavior and improve team processes. in order to do this, the agents need to understand (and perhaps contribute to) the dialog that takes place between their teammates. this case study focuses on the third of these five experiments (asist study 3), in which teams consisting of three humans and an ai advisor must work together to rescue as many victims as possible within a limited time. the mission takes place in a collapsed office building simulated in minecraft. the human players participate remotely, communicating with each other using voice chat. @xcite the goal of our event extraction component is to detect specific team coordination-related events, and relay them to ai agents, who then update their understanding of the state of the team members and the mission. the event extraction component is embedded into a larger architecture, which is described in our preregistration document (pyarelal, 2022, 5). critically, the actual design of these missions is subject to change with little notice. there is no training data for supervised approaches, and the specificity of the domain means that many opendomain approaches are unsuitable. for these rea-sons (described further in § 2) we employ a rulebased approach for our event extraction component. this allows us to rapidly pivot and adapt to changes in requirements, as well as encode details of the specific domain. in this setting, this work provides these key contributions: in this way, adaption is straightforward and endlessly repeatable. while complex structures are certainly possible with ml approaches, they are not what is supported by zero-shot approaches and we simply do not have annotated datasets of the necessary size to train such a model. ## related work rule-based approaches to event extraction (ee) have a long tradition in previous work (e.g., appelt and onyshkevych, 1998; cunningham et al., 2002; levy and andrew, 2006; hunter et al., 2008; valenzuela-escárcega et al., 2018; sharp et al., 2019). these rules were written over a variety of language forms, from surface to syntactic or semantic structures. here we use rules that combine surface and syntactic forms (valenzuela-escárcega et al., 2015), to allow for richer representations while also mitigating the effect of parsing errors resulting from imperfect transcriptions from the automated speech recognition system. that said, much of the recent literature is on machine learning approaches to ee (nguyen et al., 2016b; liu et al., 2018; sha et al., 2018; wadden et al., 2019; du and cardie, 2020b; nguyen and nguyen, 2019; xiang and wang, 2019, inter alia). much of this work, however, relies on supervision, a luxury we do not have in this setting. with the rise of large pretrained neural models, there are now several approaches to common natural language processing tasks that do not train or fine-tune a model, but instead use prompting to glean desired information from the knowledge already contained in the model (e.g., wei et al., 2021; liu et al., 2021; min et al., 2021). this category of approach is desirable in scenarios such as ours, where annotated data is unavailable and would be difficult to produce at scale. for this reason, we compare our rule-based approach to a zero-shot approach based on prompting. in § 7, we show that for our scenario, the rule-based approach performed better and was able to produce richer representations. ## approach our rule-based ee system uses the odin (valenzuela-escárcega et al., 2016) event extraction framework, which consists of an expressive, declarative rule language and a runtime system for applying the rules. more specifically, we have an odin rule grammar that extracts two broad types of events from natural language: (i) simple events that do not have arguments, foot_4 and (ii) complex events, that can take other events as arguments. each event is associated with a unique span of text and is assigned a label by the rule that extracts it. the event labels we are using are organized into a hierarchical ontology. if a rule assigns a label to an event from this ontology, the parents of that label in the ontology are also assigned as labels for that event. @xcite our system currently contains 420 active rules. these map to a total of 238 event labels, including both parent labels as well as the labels for the events we intentionally target. @xcite for example, if our system detects a event label for a specific room on the map (example: "a4"), it will output that specific event label and all its parent labels like so: "@xmath0. @xcite this allows us to look at outputs at any level of granularity and define event arguments at any level of granularity. that is, we can define an event that only requires an "a4" label as a possible argument, or else we can define an event that takes a "location" event label as a possible argument. odin's support for nested patterns allows the user to implement recursive passes on the data, as well as specifying at which pass the system should match against which specific rules. for our purposes, we use this to first look for simple events only, and then on later passes we look for complex events. the passes are recursive because prior extractions are part of the input for later passes. for example, a "room" event that was extracted in pass 3 can be an argument of a "search" event in pass 6, which can in turn be an argument of an "instruction" event in pass 12. figure 1 shows a visual representation of a sample utterance and the events extracted from it. the code, rules, and documentation for our approach are publicly available on our github repository. 9 ## data in order to quantitatively evaluate our approach and highlight shortcomings, we annotated an evaluation dataset foot_8 of in-mission dialog transcriptions for several key team coordination-related events. the dynamic nature of the experiments in asist means that annotations for asist study 3 cannot be used as training data for asist study 4 and asist study 5. for this reason we (a) only annotated enough for evaluating our approaches (rather than training), and (b) only annotated for key events, which were more likely to remain central, even as the experimental paradigm shifts. our annotated data is drawn from live mission dialog and constitutes a subset of the asist study 3 dataset (huang et al., 2022b). the dialog is transcribed in real-time using the google cloud speech automatic speech recognition (asr) system. we do not manually correct the transcription errors, as we are interested in exploring what can be achieved by our ee system in a realistic, live scenario. due to the nature of its origin, the utterances are often messy and grammatically incorrect. further, they contain filler phrases, repetitions, and interruptions. for example: okay okay yeah so many patients i need picked up was marker with the sos as mentioned above, we chose a set of 20 event labels to evaluate (see appendix for detailed explanations), consisting of labels we considered most important or interesting to us in the context of the asist program, and thus more likely to stay relevant. specifically, we selected: figure 1 : visualization of extractions produced by our rule-based system. note that some events are simple events without arguments, while others are complex, such as the "agree" and " sight" events respecitvely. additionally, events can nest, as with the "deliberateplan" event, which takes the "moveto" event as an argument. finally, note that the utterances that serve as input are often not grammatically well-formed. of our 20 labels, nine are simple events (entities) and eleven represent complex events. we tasked annotators with annotating 3686 utterances of game dialog for the labels in our set. specifically, we asked that for each event, they (a) mark the span of all arguments, and (b) label the event as a whole. we did not ask them to indicate the labels of arguments themselves. concurrently, we also annotated a subset of the same data internally for precision alone @xcite . this precision-based evaluation was done to gain more fine-grained insight into the arguments selected by our event labels. ## evaluation we evaluate our rule-based approach at two different levels of granularity: coarse and fine-grained. for the former, we compare the performance of our approach to that of a zero-shot classifier (see § 6.1). for the coarse-grained evaluation, we evaluated whether the correct labels were assigned to utterances, without checking for argument structure or spans. a given utterance may contain multiple events with the same label. @xcite for simplicity in this evaluation, and to streamline comparison with the zero-shot approach, here we evaluate extractions on a presence-only basis. that is, if the annotator assigned one "victim" label to an utterance, and our ee system assigned two "victim" labels to the same utterance, we consider this correct for the purposes of evaluation. for the fine-grained evaluation, we annotated a subset of our data for the precision of the argument structure of our outputs. we only consider outputs that we had already annotated as correct. if all their arguments match for the correct event, we consider the extraction as a true positive. @xcite during our evaluation process, we considered the code-base "frozen". we did not make any adjustments to our system based on insights from the annotation or evaluation process until the evaluation was complete and all data was gathered. in @xcite for example, a player might use the term "victim" multiple times in a single utterance. ## conclusion here we presented a case study on extracting team coordination events from natural language dialog. this dialog consists of live communications, which suffer from mistranscriptions, gap fillers, and interrupted or truncated utterances. while supervised neural approaches make up much of the recent literature for event extraction, these approaches become infeasible for this task due to the lack of training data. further, the rapidly changing requirements for what should be extracted, as well as the need to embed domain-specific knowledge, led us to implement a rule-based approach. recently, large-scale pretrained language models have made zero shot and/or prompting-based approaches a convenient go-to for strong baselines, as these approaches make knowledge gained through large-scale pretraining accessible to various tasks. for this reason, we compare our rule-based method to a zero-shot classifier approach. we showed that in our case, our rule-based method out-performed the classifier for all event types we considered. furthermore, the events output by our rules contain rich structure that can be used by downstream components for inference.