Requesting clarifications with speech and gestures (2021)

Abstract: In multimodal natural language interaction both speech and non-speech gestures are involved in the basic mechanism of grounding and repair. We discuss a couple of multimodal clarifica- tion requests and argue that gestures, as well as speech expressions, underlie comparable paral- lelism constraints. In order to make this precise, we slightly extend the formal dialogue frame- work KoS to cover also gestural counterparts of verbal locutionary propositions.

## introduction detailed taxonomies of verbal clarification requests (crs) already exist @xcite and accounting for these motivate theories of grounding and clarification interaction such as @xcite @xcite , which provide wide coverage thereof. although there exists some corpus-based and experimental work on multimodal repair @xcite @xcite , detailed taxonomies are yet to be developed, nor formal accounts thereof. in this paper we consider how to account for the multimodal versions of one of the commonest types of clarification request dubbed reprise fragments by @xcite . clarification requests play an important role in semantic methodology @xcite and in the construction of dialogue systems @xcite . @xcite argue in detail that these exhibit significant syntactic and phonological parallelism with their source, as exemplified in (1a); concretely their claim is that the intended content reading ('what do you mean by . . . ') requires segmental identity with the source. a similar condition mutatis mutandis seems to be the case for gestural ones (2): (2a,b) involve clarifying a body movement (the former from example (1), @xcite , the latter two concern laughter, either with respect to content or in the latter case clarifying the manner of laughter ((2e) is from fig. 1 of healey et al., 2014, 26): foot_0 (1) a. (i) a: do you fear him? b: fear? (= what do you mean by 'fear' or are you asking if i fear him) / #afraid? (ii) a: are you afraid of him? b: afraid? (= what do you mean by "afraid"? or are you asking if i am afraid of him) / #fear? b. a: are you afraid of him? b: afraid? (= what do you mean by "afraid"?) (2) a. b: you have to move your legs like this [moves right hand up and down in a wave-like manner]. a: [moves right hand up and down in a wave-like manner, raises eye-brows] b. . . . and that movement really cracks your back c. what's that? you do that and someone pulls? d. a: i hear you're busy laughter [= little giggle]. b: laughter ? (= low arousal laughter with rising contour). (attested example) e. was it kind of like [h:o?]= [h:hhh] clarification requests also occur on larger time scales, as is evinced in figs. 1 to 3. the example is taken from the speech and gesture alignment corpus saga @xcite , which is a multimodal corpus of route direction dialogues. the example is about a section of a route where the route follower has to enter a park and walk around a pond, but not completely, just to three quarters. the route section is described by the route giver in fig. 1 . it is put to clarification by the addressee (route follower) in fig. 2 . abstracting over perspective, the moving around movement is more or less kept constant, but modelling the pond is changed from a gesture hold to a drawing on the back of the hand. the route giver subsequently corrects the clarification by a path drawing on the addressee's back of hand in fig. 3 . we show how to extend existing notions of conversational context and representation of speech multimodally to account for such cases. the basic extensions to the formal framework introduced in the following section are (i) multi-tier partiturs for capturing signals on different channels, (ii) a classification of gesture events on the tiers, and (iii) an anaphoric multimodal clarification rule requesting feedback concerning a previous multimodal fragment. ## background our account is formulated within type theory with records (ttr, @xcite . ttr is a formal semantics framework based in the proof-theoretic, intuitionistic mathematics of @xcite . the reason for using a formal framework is that it enables researchers working on semantic phenomena in a scientific, precise manner. this is possible since the interpretation of types and structures used can be fixed in models-for such a denotational interpretation of ttr see @xcite . 2 although traditionally mainly applied to the compositional semantics of sentences, semanticists working on dialogue soon developed conversation-oriented extensions (just think of the content of particles such as hi! or yes or highly normative patterns such as question-answering.) however, classifying (multimodal) natural language utterances is not always a binary affair (think, e.g., of vagueness). to this end, there are probabilistic interpretations of ttr 2 the semantic status of natural language processing (nlp) remains unclear, to say the least, as recently pointed out by @xcite . however, theoretical work such as the one developed here can of course trigger nlp applications, where, e.g., theoretically derived labels are automatically annotated on large(r) scale data. @xcite . although we could render our discussion in probabilistic terms, foot_1 we refrain from doing so since this paper is not concerned with probabilistic phenomena as such and this keeps representations simpler. ttr integrates logical techniques such as the lambda calculus and the expressiveness of feature-structure like objects (namely records and record types). a typing judgement a : t is true iff object a is of type t . types constructed from n-ary predicates (@xmath0. thus, if a 1 : t 1 , a 2 : t 2 (a 1 ), . . . , a n : t (a 1 , a 2 , . . . , a n-1 ), then the record on the left in (3) is of the record type on the right in (3): the notation [@xmath1: t ] represents a manifest field @xcite . it is a notational convention for a singleton type t a , where for any b, b : t a iff @xmath2. merge types correspond to unification in featurestructure formalisms. a merge '∧ . ' is exemplified in (4): drawing on work of @xcite , ttr comes with a string theory of events. for three events e 1 , e 2 and e 3 , the string e 1 e 2 e 3 represents a course of events, namely the succession of e 1 , e 2 and e 3 , in that order. the notation e 1 e 2 e 3 is an abbreviation for a time-indexed record: if e 1 : t 1 , e 2 : t 2 and e 3 : t 3, then e 1 e 2 e 3 : t 1 t 2 t 3 -the type constructor ' ' builds string types out of types. in order to exploit feature structure expressiveness in string types, a string of record types can be build by the same means, but is notationally enclosed in brackets. figure 3: 'du fährst noch weiter rum.' (you drive around even more.): stretched index finger is moved around the virtual pond. making use of ttr, the simplest model of context, going back to @xcite is one which specifies the existence of a speaker, addressing an addressee at a particular time. this can be captured in terms of the type in ( 6 ). however, over recent decades it has become clearer how much more pervasive reference to context in interaction is. the visual situation is a key component in interaction from birth (see tomasello, 1999, chap. 3). expectations due to illocutionary acts-one act (querying, assertion, greeting) giving rise to anticipation of an appropriate response (answer, acceptance, counter-greeting), also known as adjacency pairs @xcite . extended interaction gives rise to shared assumptions or presuppositions @xcite , whereas epistemic differences that remain to be resolved across participants-questions under discussion are a key notion in explaining coherence and various anaphoric processes @xcite . these considerations among several additional significant ones lead to positing a significantly richer structure to represent each participant's view of publicized context, the dialogue gameboard (dgb), whose basic make up is given in (7), following the recent version of the dialogue semantic framework called kos including mood described by @xcite : here facts represents the shared assumptions of the interlocutors-identified with a set of propositions. vis-sit represents the visual situation of an agent, including his or her focus of attention (foa), which can be an object (ind), or a situation or event (sit). the remaining fields concern locutionary and illocutionary interaction: dialogue moves that are in the process of being grounded or under clarification are the elements of the pending list; already grounded moves are moved to the moves list. within moves the first element has a special status given its use to capture adjacency pair coherence and it is referred to as latestmove. the current question under discussion is tracked in the qud field, whose data type is a partially ordered set (poset). mood tracks public displays of emotion, crucial for inter alia laughter and smiling @xcite . the evolution of context in interaction is described in terms of conversational rules, mappings between two cognitive states, the precond(ition)s and the effects. some examples of such rules are given in ( 8 ): (8)a. ask qud-incrementation: given a question q and ask(a,b,q) being the latestmove, one can update qud with q as maxqud.       pre : q : question @xmath3,addr,q) : illocprop effects : @xmath4,pre.qud : poset(question)       b. assert qud-incrementation: a straightforward analogue for assertion of (8a): given a proposition p and assert(a,b,p) being the latestmove, one can update qud with p? as maxqud.       pre : p : prop @xmath5, addr, p) : illocprop effects : @xmath6, pre.qud : poset(question)       as emphasized by @xcite and by work in conversation analysis (ca; @xcite grounding and clarification interaction are important structuring processes in interaction. in @xcite these are modelled as a process triggered by awareness of an utterance event u and the attempt to instantiate the fields of an utterance type t u emergent from parsing and resolving u. the pair of u and t u is referred to as locutionary proposition locprop. this is a special kind of (austinian) proposition-records of type sit : rec sit-type : rectype @xcite 4 -where sit is an utterance event and sit-type the type of a grammatical sign. this allows inter alia access to the individual constituents of an utterance. @xcite and @xcite show how to account for the main classes of crs using rule schemas of the form "if u is the interrogative utterance and u0 is a constituent of u, allow responses that are co-propositional 5 with the clarification question cq i (u0) into qud.", where 'cq i (u0)' is one of the three types of clarification question (repetition, confirmation, intended content) specified with respect to u0. thus, the schema 'if u is an utterance spoken by a and u0 is a constituent of u, allow responses that are co-propositional with "what did a mean by u"' can be formulated as in (9): the issue q0, what did a mean by u0, for a constituent u0 of the maximally pending utterance, a its speaker, can become the maximal element of qud, licensing follow up utterances that are copropositional with q0. assuming a propositional function view of questions, copropositionality allows in propositions from the range of range(q0) and questions whose range intersects range(q0). since copropositionality is reflexive, this means in particular that the inferred clarification question is a possible follow up utterance, as are confirmations and corrections, as exemplified in (10a-c). 4 on this view, a proposition @xmath7: t -the situation s is of the type t . ## partiturs in order to utilize the information state update semantics of kos for analysing multimodal discourse, we add extra structure to the utterance events by incorporating tiers. tiers can be likened to different instruments on a musical score: a partitur. 6 we represent partiturs as strings of multimodal communication events, which is a temporally ordered sequence of types. one can think of strings in term of a flip-book: a dynamic event is cut into slices, and each slice is modeled as a record type. such string types @xcite are notated in round brackets: (11) partitur :=        e : (        e speech : phon e gesture : trajectory e @xmath8: rectype e head : headmove e face : faceexpr the progressive unfolding of sub-events on the various tiers in time gives rise to incremental production and perception. formally, this is indicated by the kleene plus (' + '): the string type in (11) classifies events which consists of a sequence of multimodal communication signals. hence, partiturs provides a formal means for describing crosstier interaction. in order to model one sort of multimodal integration we make use of the account of speech-gesture of @xcite , respectively its ttr reformulation @xcite . speech-gesture integration on this account is modelled in terms of a speechgesture ensemble @xcite , where a gesture (g-dtr) from tier e gesture attaches to a phonetically marked affiliate (aff; schegloff, 1984) from speech (s-dtr, tier e speech ). thus, multimodal integration of this sort is constrained by both temporal alignment and phonetic-kinematic interface (cf. also @xcite . semantic integration is formally governed by a imagistic feature called conceptual vector meaning ("cvm"). cvm draws on abstract motion perception from psychophysics @xcite and can in semantics formally spelled out in terms of vector-based representations of shapes, movements, orientations, or object axes within the vector space algebra of @xcite . the basic integration scheme is given in ( 12 ): (12) sg-ensemble    @xmath9.phon : phon @xmath10.cat : syncat @xmath11.traj ∧ . s-dtr.cont.cvm    s-dtr    phon.accent : marked cat : syncat cont : semobj    g-dtr @xmath12: sign traj : vec the underlying rationale of ( 12 ) is that both a gesture movement and a cvm value is a trajectory that is mathematically described as a sequence of vectors in three dimensions (r 3 ; or r 4 if the temporal dimension is explicitly built in). drawing on work in gesture annotation, gestures are represented in terms of their kinematic features, giving rise to a 'phonetic' gesture representation. for example, moving the wrist rightwards, back (i.e., towards the body of the gesturer), and leftwards in a rectangular manner ('line')path : line @xmath13: move a cornered, horseshoe-shaped trajectory ' ' is displayed. via a translation procedure from gesture representations onto vector representations, the abstract trajectory in (13) is obtained @xcite . spatial predicates also carry trajectory information as part of their cvm feature. the vector sequence from ( 13 ) is part of the lexical entry of the adjective u-shaped (it modifies a nominal, whose content is an individual). ( 14 ) since the gesture's trajectory and the adjective's cvm value are compatible, both can merge into a sg-ensemble. 7 abstracting away from concrete movements to abstract vector representations seem to provide a format that is appropriate for gestural parallelism constraints, as will be discussed in sec. 4. an example involving the 'u-shape' gesture is used by @xcite : the house [has a rectangular] shape. the noun phrase the house has a rectangular shape is accompanied by a rectangular shape gesture which temporally overlaps the bracketed portion of speech. this tier-crossing utterance is incrementally processed by a multimodal chart parser @xcite @xcite . the string chart in (15) represents the state after having processed the house has and the gesture's preparation phase. due to this input, a vp rule (e 9 ) and a gesture integration rule (e 10 ) have been triggered, but are still pending: note that a multimodal ensemble-e 10 in ( 15 ) and ( 14 )-differs from phrasal constructions usually described by grammar: while the constituents of phrases are serialized (as captured in the string type 'e' in (15)), constituents of ensembles usually co-occur. in terms of locutionary propositions, the structure of an ensemble-consisting of a manual gesture and speech-is as in ( 16 ): in contrast to the 'horizontal' chart parsing edges represented in terms of string types in the preceding incrementally growing partiturs, the daughters of multimodal ensembles are combined via 'vertical' edges. such edges are defined in multichart parsers which have been developed exactly for the purpose of processing multimodal input @xcite . we notate tiercrossing bindings on the level of utterance events (where an utterance comprises speech and gesture) in terms of the reserved label e sync -such combined representations are object of at least one class of gestural clarifications. ## gestural clarification: the case of reprise fragments in this section we show how to modify an existing account of speech reprise fragments with minimal additions, though important empirical questions about the unity of this type of clarification request remain. the analysis proposed by @xcite for this class of reprise fragments involves two components: with respect to the former task we need to generalize the condition @xmath14.sit-type.phon in (17) so that it can apply to gestures, laughs and their combinations with speech. the most obvious generalization would be to require type identity with respect to form on all tiers. however, this will not work because in all cases small but important divergences actually need to apply. in the case of speech the identity is segmental identity, but not with respect to the speech contour (where the reprise is typically lh), whereas in the case of gesture reprises the face is required to involve a frown (in the facs system @xcite a combination of a(ction)u(nits) 1 and 4 @xcite .). indeed it seems like a repetition which involves total form identity such as repetition of an utterance that is already bearing an lh contour or repeating a frown cannot be understood as clarification requests-they cannot be understood as clarifying the clarification requests (which could be achieved by saying e.g., 'what do you mean . . . '): in both cases, then, one needs to leave a channel free, presumably to express interrogative force. hence, the most straightforward way to achieve this generalized parallelism condition is simply to specify the facial form as identity modulo specification of aus 1 and 4 and the speech form as identity modulo intonation. an additional question is whether or not multimodal reprises require all channels to be reactivated, as exemplified in ( 21 ). we hypothesize that only the complete reprise can communicate a 'what do you mean' content, whereas the other reprises are understood as confirmations. however, clearly this requires experimental investigation. (21) a: i don't care + shrug. b: you don't care + shrug + frown?/ you don't care?/shrug + frown for now we will postulate a generalized utt-anaph type, building on (16) (22) mm-utt-ana-@xmath15: rc : mmprop formtype : type c1 : quasi-identical(rc.syn,form-type) syn : formtype @xmath16.mm-event.e sync : rec         why can a frown give rise to a clarification question in this context? we assume, following @xcite , who in turn build on proposals of @xcite ; @xcite , that frowns communicate the emergence of a problem in interaction, more specifically involve the frownable giving rise to a question, which can indeed be spoken: this is backed by entries on eyebrow raise and, even stronger, eyebrow cock in the nonverbal body dictionary, which are described as signalling surprise, excitement, or general disbelief ( http://bodylanguageproject.com/ nonverbal-dictionary/ , accessed april 27, 2021). eyebrows are also used as question markers in sign languages (e.g. @xcite . there different kind of eyebrow movement are correlated with different types of sentences (e.g., yes-no vs. wh; see @xcite for a particular clear overview of eyebrow use in brazilian sign language questioning). how to package this to attain a construction akin to ( 18 )? there seem to be two options: assume that there is a single reprise fragment construction with certain components that are optional. on this line all instances spoken and purely gestural involve frowning with an utterance anaphora constituent involving phonological or gestural parallelism. the other option is to assume two subtypes of such a construction, a spoken one which involves an lh tone sequence, and a gestural where the interrogative force is driven by the frown. choosing between these options requires a detailed experimental study, which we leave for future work. for concreteness we offer in (24) a sketch of the former strategy: a precise semantic analysis along these lines of the discourse functions of gestures in multimodal interaction is attained (for a related work on the socalled what are you talking about face see @xcite . such analyses are needed in order to understand and model tier-crossing coherence in natural language processing, in both artificial and human agents. crs are a key interactional competence in this respect. ## conclusions clarifications requests are an important dialogical resource for seeking mutual understanding and driving conversational interactions. however, in faceto-face dialogue crs extend to the full range of verbal and non-verbal signals. we provided some data illustrating the phenomena at stake and introduced the basic ingredients to develop multimodal clarifications for linguistic theories. this work fills in particular two explanatory gaps left by current multimodal studies, namely (i) projecting (non-emblematic) gestures to illocutionary acts, and (ii) connecting gestures to the basic dialogue dynamics of grounding and repair.