A Two-Level Interpretation of Modality in Human-Robot Dialogue (2020)

Abstract: We analyze the use and interpretation of modal expressions in a corpus of situated human-robot dialogue and ask how to effectively represent these expressions for automatic learning. We present a two-level annotation scheme for modality that captures both content and intent, integrating a logic-based, semantic representation and a task-oriented, pragmatic representation that maps to our robot’s capabilities. Data from our annotation task reveals that the interpretation of modal expressions in human-robot dialogue is quite diverse, yet highly constrained by the physical environment and asymmetrical speaker/addressee relationship. We sketch a formal model of human-robot common ground in which modality can be grounded and dynamically interpreted.

## introduction the interpretation of modal expressions is essential to meaningful human-robot dialogue: the ability to convey information about objects and events that are displaced in time, space, and actuality allows the human and robot to align their environmental perceptions and successfully collaborate @xcite . as an example, if a robot is sent to a remote location on a search and navigation mission, modally interpreted expressions such as "tell me what you see" (uttered by the human) and "i can't see because of smoke" (uttered by the robot) are vital to information exchange. similarly, a robot that has abilities to navigate obstacles (for example, by jumping or using lidar) can inform the human of this. the learning of modal expressions for automatic understanding and use nevertheless presents a conversational paradox: while these expressions serve to communicate and align world knowledge, there is no obvious manner to ground them in the shared environment. whereas objects and actions can be pointed to or modeled for grounded learning, modal expressions are grounded in the linguistic signal itself. nevertheless, a basic understanding of modal meaning would allow non-human agents to reason about the possible uses of objects and better assess how certain actions and behaviors impact the task at hand. in this paper, we document the range and nature of modally interpreted expressions used in human-robot dialogue with the goal to make the interpretation of such expressions easily automated in the future. we hypothesize that certain readings and scope preferences for modal operators are more salient in human-robot dialogue because of the unique makeup of the common ground @xcite . we provide a mapping from formal semantic theories of modality related to participant beliefs and updates of the common ground @xcite , to a practical model of speech acts that translates into robot action for search and navigation task-oriented dialogue and an automated nlu and nlg system @xcite . this mapping is formalized in an annotation scheme in which the use of modal expressions is mapped to their effect in dialogue, providing a model for the robot to learn the meaning of modal expressions @xcite . our annotation task reveals surprisingly high inter-annotator agreement for a complex scheme; results indicate that our data is highly repetitive in the natural language used, and yet the interpretation of modal expressions is quite diverse and worth investigating further to foster effective human-robot communication in situated, task-oriented settings. the paper is structured as follows. in section 2 we motivate our annotation of modality and introduce the scout corpus; we then situate formal semantic theories of modality in the context of human-robot dialogue. we describe our annotation scheme in section 3, which covers both the type of modality used in an expression, and the speech act the expression conveys. we describe our results in section 4, discussing implications for modal interpretation in human-robot dialogue and some linguistic issues that arose during the annotation process. in section 5 we consider the implications of our results for a theory of modality and common ground in human-robot dialogue, before concluding in section 6. ## a two-level annotation scheme the motivation for a two-level annotation scheme comes from the need to bridge formal semantic theories of modality and its interpretation with models that are actionable in the context of human-robot dialogue and adequately model the discourse. in this section we discuss the development of our annotation scheme, drawing on both fine-grained annotation of modality (section 3.1) and the identification of speech acts specific to human-robot dialogue (section 3.2). we present our final annotation scheme in section 3.3. modal value based on... example from scout epistemic ...belief or knowledge. i think you are more familiar with objects than i am. (@xmath0...what someone or something can do. can you manipulate objects? (@xmath1...what the rules, standards, or social norms state. i need your help to decide which are important. <@xmath2...someone's wishes or desires. how far would you like me to turn left? (@xmath3...the achievement of goals. i can move closer to take a picture. (@xmath4...syntactic form and imperative mood. please be aware of lag time. (@xmath5 ## annotation task our goal for the annotation task was two-fold: (i) to provide coverage of the data to quantitatively assess the kind and frequency with which modal expressions are used and interpreted by speaker type; and (ii) to qualitatively assess instances where modal usage is unexpected. this second goal is situated within a larger goal of understanding and automating the interpretation of scope in human-robot dialogue. four annotators were trained to apply the annotation scheme following the annotation guidelines and with two example annotated transcripts. each annotator annotated 70 experimental transcripts, of which 16 transcripts overlapped with one of the other three annotators. in total, 248 transcripts were annotated: 32 by two annotators, and the remaining 216 by a single annotator. annotators were instructed to only annotate the left conversation floor ( to evaluate our annotation scheme, we calculated a number of inter-annotator agreement metrics on the 32 transcripts annotated by two annotators. first, we calculated the proportion of annotations for which the annotators agreed on the target. among those annotations where the annotators agreed on the target, for each pair of annotators, we calculated the string overlap foot_3 between the scopes identified by each ## towards a formal theory of modality in human-robot dialogue here, we sketch the beginnings of a formal theory of modality in human-robot dialogue built upon our annotation findings in section 4. as stated before, this work takes a step towards automating the interpretation of frequently ambiguous expressions in context and mapping this interpretation to actionable representations in the human-robot context. ## conclusion in this paper, we present a two-level annotation scheme for modality as used in situated human-robot dialogues relating to search and navigation. our annotation scheme captures both the semantic content of modal expressions as well as their pragmatic function relevant to speaker intent in discourse. results from our annotation task demonstrate that our annotation scheme is valid and expressive, as well as both practical and transparent; it also gives us novel insight into the interaction between modality and illocutionary force in our setting. our work can be extended to future, automated pipelines for human-robot dialogue that incorporate modal expressions within a formal common ground. interpretation commitments & obligations examples 1. question speaker (s) committed to desire to know answer; addressee (a) obliged to respond to question do you see foreign writing? (ability) are you able to move that orange cone in front of you? (ability) 2. assertion s committed to a state of affairs i think you are more familiar with shoes than i am. (epistemic) i'm not able to manipulate objects. (ability) 3. offer s committed to feasibility of plan of action; a obliged to consider action and respond would you like me to take a picture? (bouletic) i will move forward one foot, ok? (teleological) 4. promise s committed to feasibility of plan of action and obliged to do action i will send a picture. (teleological) i'll give you feedback on what i'm doing. (teleological) 5. command s committed to desire for a to do something and feasibility of action; a obliged to do action back up three feet. (imperative) help! (imperative) 6. open-option s committed to feasibility of action(s) if you describe an object, you can help me learn what it is. (ability) you can tell me to move a certain distance or to move to an object. (ability) 7. request s committed to desire for a to do something and feasibility of action; a is obliged to consider action and respond can you repeat that? (ability) i need your help to find shoes. (deontic) 8. accept/reject s committed to a state of general acceptance or rejection i think you are correct. (epistemic) yes please proceed. (imperative) 9. greeting s committed to recognizing presence of a and willingness to interact hello! 10. gratitude s committed to state of gratitude thanks, teammate! 11. regret s committed to state of regret sorry robot i meant west. (epistemic) 12. judgment s committed to evaluative stance the containers look like something that could be moved. (ability) possible equipment or storage space they might need for something for gathering. (teleological) 13. mistake s committed to acknowledging error i mean twenty degrees to the left. (teleological) actually i meant turn left fifty degrees. (teleological) 14. hold floor s committed to holding conversational floor for continued speech (not found in our data)