Reference in Team Communication for Robot-Assisted Disaster Response: An Initial Analysis (2020)

Abstract: We analyze reference phenomena in a corpus of robot-assisted disaster response team communication. The annotation scheme we designed for this purpose distinguishes different types of entities, roles, reference units and relations. We focus particularly on mission-relevant objects, locations and actors and also annotate a rich set of reference links, including co-reference and various other kinds of relations. We explain the categories used in our annotation, present their distribution in the corpus and discuss challenging cases.

## introduction we present the findings of an initial analysis of contextual reference phenomena in team communication in robot-assisted disaster response. disaster response teams operate in high risk situations and must make critical decisions quickly, despite partial and uncertain information. for better safety and operational capability, first responders increasingly deploy mobile robots for remote reconnaissance of an incident site. the work in this paper contributes to our ongoing effort to develop methods for interpreting the verbal communication in a response team, in order to extract run-time mission knowledge from it. mission knowledge encompasses the mission goals, which tasks have been assigned to whom, the state of their execution, the relevant points of interest (pois) and the possibly changing information about them, etc. we work on using mission knowledge extracted from the verbal team communication and integrated with information from other sources, such as the sensors carried by the robots, to provide situation awareness and teamwork assistance both during and after a mission, as described in @xcite . as part of extracting mission knowledge from the verbal team communication, it is important to identify mission-relevant objects, locations, actors, tasks, events etc. that are being referred to and the links between them. this is the goal of reference resolution. in order to get a better understanding of how this task can be performed, we analysed the corpus of robot-assisted disaster response team communication from the tradr project (tradr project website, 2020), @xcite . the corpus contains the communication in teams of first responders who are using ground and airborne robots to explore an area, searching for victims and hazards, to carry out measurements and gather samples in the aftermath of an industrial incident, such as a fire or explosion. in the first phase of reference phenomena analysis, the results of which we present in this paper, we focused on the references to and links between mission-relevant objects, locations and actors. we aimed to gain insight regarding the kinds of reference cases in the data, their distribution and the challenges for reference resolution. we annotated the data in order to systematically capture the various cases and be able to access them later for deeper analysis. our aim was not to create an ultimate annotated resource and/or a novel annotation scheme. we designed the annotation scheme specifically for our analysis, and it was evolving as the annotation progressed. the present paper reports our findings and indicates what our reference resolution system would need to deal with. in section 2 we overview existing approaches to co-reference and anaphora annotation in text and dialogue. section 3 gives more details about our data. section 4 describes the cases of reference to and links between mission-relevant objects, locations and actors that we identified and how we captured them in our annotation scheme, accompanied by typical examples as well as illustrations of some tricky cases and challenges. in section 5 we summarize and indicate our future steps. ## approaches to co-reference and anaphora annotation the study of co-reference and anaphoric relations has long tradition in linguistics, and there exist numerous approaches to annotation. they are rather difficult to systematise, due to proliferating terminologies and combinations of heterogeneous phenomena, such as different types of references, referents and relations. since a proper discussion of the similarities and differences would exceed the space we have available here, we present the relevant previous work in a close to chronological order. the first annotation scheme for anaphoric relations appeared in 1992 @xcite . in the late 1990s this field of research got a push, when the 6th and 7th message understanding conferences (muc-6, muc-7) took place. muc-7 coreference task definitions @xcite ) defined co-reference as a symmetric identity relation between two noun phrases (nps) if both of them refer to the same entity. among the researchers who worked on co-reference annotation schemes around this time are @xcite , ge (1998), @xcite . there also appeared some works investigating not only relations between entities introduced by nps, but also event co-reference and temporal relations between events, e.g. by @xcite , or @xcite . at the same time, interest in the annotation of a wider range of semantic relations emerged. among these relations is anaphoric reference. while co-reference is an equivalence relation, anaphoric reference is not -the interpretation of an anaphoric expression always depends on its antecedent. @xcite discussed the differences between anaphora and co-reference in detail. they stressed that they can coincide, but are not interchangeable, and pointed out that co-reference is not to be mixed with bound anaphora, where an anaphor relates to a generic antecedent, which does not actually refer to any specific entity. they also showed the difference between co-reference and an intensional relation between an entity and a predicative expression that refers to a whole set of entities. one of the more fine-grained approaches to co-reference annotation was presented by @xcite . aiming at creating corpora for event processing, they investigated np and event co-reference and created a co-reference annotation scheme. they introduced relations between nps (identity, synonymy, generalisation, specialisation and other) and co-reference types (np, copula, apposition, bracketed text, speech pronoun and other). cohen et al. (2017) about identity and appositive relations in biomedical journal articles, dakle et al. (2020) on co-reference in emails, wright-bettner et al. (2019) on cross-document co-reference. while all the above mentioned works mostly deal with text data, @xcite developed the mate 'meta-scheme' for anaphora annotation in dialogue. this generic scheme consists of a core scheme for annotating identity relations (co-reference) between the entities introduced by nps, and three extensions for annotating references to the visual situation, bridging and anaphoric relations involving an extended range of anaphoric expressions and antecedents. the bridging extension was later realized in the gnome annotation project @xcite . @xcite also formulated the difficulties that any designer of an annotation scheme for anaphora faces, namely, that almost every word or phrase in a coherent text can potentially be linked to something that was introduced earlier (cf. also the concept of cohesion (halliday and hasan, 1976)). speaking of anaphora, sometimes researches try to concentrate only on the identity relation between entities, e.g. @xcite or @xcite . but often anaphora is understood in a broader sense. @xcite and @xcite discussed in various types of anaphors and antecedents. zinsmeister and dipper (2010) researched the annotation of abstract (discourse-deictic) anaphora. anaphoric relations between different types of events were also studied by @xcite . @xcite , annotating the arrau corpus, introduced an anaphoric relation between single anaphoric expression and plural antecedents, as well as references to events, actions and plans. @xcite developed a discourse-level annotation scheme that covered a broad range of discourse reference properties, e.g. semantic sort, delimitation, quantification, familiarity status, and anaphoric links (co-reference and bridging). there also emerged some classifications of anaphora types and other relations. so, @xcite annotated the monroe corpus, consisting of task-oriented dialogues from an emergency rescue domain. they focused on co-referential pronouns and nps (identity relation), but also presented a classification of relations for non-co-referential pronouns with the following relation types: indexicals, action, demonstrative, functional, set, hard and dummy. @xcite distinguished three types of abstract anaphora: label anaphora (encapsulates stretches of text), which has several sub-types, situation anaphora (for linking events, processes, states, facts, propositions) and text deixis. another classification by eckart de @xcite included the following anaphora types: individual anaphors, reference to abstract objects, vague anaphors, inferrable-evoked pronouns and unmarked anaphors. all these classifications were developed with a certain corpus and task in mind, and none can be considered universal, standard and pervasive. although none of the existing classifications was entirely suitable to us, given our domain and the goal of our analysis, they were fundamentally helpful in devising our own annotation scheme. so, following @xcite , we try to differentiate between an explicit identity, when expressions are linked via a copula, and an implicit one. defining the bridging relation, we relied on works of @xcite and @xcite . our intensional relation can be traced back to both bound anaphora and intensional predicates presented by @xcite , and the notion of vague anaphor is similar to that defined by eckart de @xcite . we started the annotation effort with the aim to keep the annotation scheme quite simple, distinguishing between main types of mission-relevant entities, locations and actors, and focusing on the reference relation types. as we proceeded with the annotation, we found it necessary to extend the scheme with certain more fine-grained distinctions to capture sometimes quite special cases. the tradr corpus consists of dialogues that represent human-human team communication in robot-assisted disaster response. the dialogues were recorded during exercises on different industrial sites performed as part of the tradr project (tradr project website, 2020), @xcite . the exercises simulated situations after a industrial accident, such as fire, explosion, etc. and involved teams of firefighters using ground and airborne robots for reconnaissance. there are 15 files with dialogues in the corpus, each corresponding to a mission or sometimes a part of a mission. nine files contain dialogues in german, and six -in english. the german dialogues were recorded in 2015 and 2016, the english data is from 2017. the firefighters who took part in the 2017 experiment were dutch, and so nonnative english speakers. in total the joint corpus contains about 2,9k dialogue turns (see the tradr experiments involve teams of first responders exploring complex dynamic environments using robots, namely unmanned ground vehicles (ugvs) and unmanned aerial vehicles (uavs). the robots are used for reconnaissance, mainly to look for points of interest (pois), including victims and hazard sources, such as smoke, fire, or contamination; and check if the site is safe enough for human first responders to enter. the robots are equipped with gas detectors, a standard camera and an infrared one. pictures taken by the robot cameras can be shared among the team members. some ugvs have a mechanical arm for picking up, turning, pushing or moving objects. the team consists of operators (ugv-1, ugv-2, uav) who control the robots, a team leader (tl) and in some missions also a mission commander (mc). a mc is in charge of the whole mission and gives tasks to teams. the tl distributes the tasks between the operators, coordinates their actions and reports to the mc (if present). the operators use robots to perform the tasks assigned to them and report to the tl about the results or possible difficulties. the team members use a shared situation awareness interface, consisting of a digital map on which pois are marked and robots' positions are displayed; a repository of shared photos made with the robot camera; and in 2017 also a task list which the tl can manually edit. the team communication in the tradr scenarios has the following characteristics: ## annotation and analysis of reference in the tradr corpus the analysis we present aims to gain initial insight in the kinds and distribution of references to entities and relations between them in the tradr corpus, as a preparatory step before developing reference resolution modules as part of the team communication interpretation in our system @xcite . in this section we explain the categories that we distinguished in the analysis, show their distribution in the tradr corpus and discuss the challenges we encountered during annotation. we used the webanno tool (eckart de @xcite to perform the annotation. originally, only one annotator, the first author, performed the annotation, under the guidance of the second author. all spurious cases were discussed by both authors and the annotation was updated based on the decision. we adjusted and extended our annotation scheme in the process. we did not involve multiple independent annotators, because our aim was mainly to get an overview of the reference resolution issues. to test the reliability of the resulting annotation scheme, another person annotated a small subset of the corpus, consisting of one dialogue, which contained 57 utterances. we measured inter-annotator agreement using cohen's kappa following @xcite . we obtained a kappa score of 0.704 for the entities layer, 1.0 for comments, 0.895 for roles, 0.573 for reference units and 0.845 for reference links. this shows good agreement, except for reference units. ## conclusions and outlook we presented a preliminary analysis of reference phenomena in a corpus of team communication for robot-assisted disaster response, done in preparation for developing reference resolution modules for a system that interprets such team communication to extract run-time mission knowledge and use it for various forms of teamwork assistance. our annotation scheme has separate layers for mission-relevant entities, roles of actors, reference units with links between them, as well as comments for special cases. among the mission-relevant entities we focused on objects, locations and actors. although these constitute the majority of objects referred to, there is a large number of other entities that remain to be classified in more detail. for the sake of reference resolution we found it important to distinguish between objects and locations that are known to exist, and those that are potential/hypothetical or undefined. this is however very difficult to do during annotation. moreover, information about mission entities evolves during the dialogue, and this creates challenges for co-referential links. another interesting challenge is the double reality representation, which means that mission objects in the physical reality and those displayed in a digital map are mostly referred to interchangeably, but sometimes need to be distinguished. in this regard we plan to review the literature on visual co-reference resolution as a next step. our analysis shows that a content representation using slots and fillers, as is commonly done in dialogue systems, clearly does not suffice for this domain, a proper discourse representation is required. as for referential links, basic anaphora together with identity relationships dominate. bridging is next, and then there are various different cases which are not that frequent but quite tricky for reference resolution. analyzing different kinds of bridging in more detail and properly describing the other kinds of links remains a topic for our future work. the annotated corpus is currently being used for testing existing co-reference resolution models, including the allennlp model @xcite , neuralcoref by huggingface @xcite and the corenlp framework @xcite . the results of these experiments will help determine our future steps for reference resolution.