Towards the ISO 24617-2-compliant Typology of Metacognitive Events (2021)

Abstract: The paper presents ongoing efforts in design of a typology of metacognitive events observed in a multimodal dialogue. The typology will serve as a tool to identify relations between participants’ dispositions, dialogue actions and metacognitive indicators. It will be used to support an assessment of metacognitive knowledge, experiences and strategies of dialogue participants. Based on the mutidimensional dialogue model defined within the framework of Dynamic Interpretation Theory and ISO 24617-2 annotation standard, the proposed approach provides a systematic analysis of metacognitive events in terms of dialogue acts, i.e. concepts that dialogue research community is used to operate on in dialogue modelling and system design tasks.

## introduction daily life is replete with determinations about the reliability of our own thoughts and feelings as well as attributions about the thoughts and feelings of others. these metacognitive capacities underlie cognitive and social adaptation, influence decisionmaking, can enhance self-efficacy. metacognition enables cognitive control needed for people to anticipate the future task demands, improves learning and performance on complex memory tasks, knowledge transfer and task switching @xcite . cognitive models of metacognitive processes, when integrated into human-computer dialogue, transform the dialogue system from a reactive dialogue participant into a proactive learner, accomplished multi-tasking planner and adaptive decision maker @xcite . metacognitive capabilities of existing interactive systems, even of complex smart learning environments @xcite , are still rather limited so as metacognitive strategies used. to exploit the full potential of efficient metacognitive support in a dialogue system, big multimodal data samples are required to reliably identify metacognitive states accounting for a complexity of multidimensional contingencies between tasks, performed actions and participants' cognitive and emotional dispositions. an elaborate computational model of (meta)cognitive states calls, in the first place, for a typology of metacognitive events -reflexive activities that express the sender's mindful awareness of own and others cognitive processes, e.g. checking out and verification of attention, recognition, understanding, evaluation and regulation of content, thought processes, attitudes, preferences, assumptions and emotions. metacognitive events should be computable/learned from a range of low level multi-sensory and psycho-physiological indicators (markers). methods are required to transform multimodal data in a meaningful way to enable appropriate measurements of metacognition, adaptive decision-making and efficient coordination of multiple dialogue tasks. the main goal of the presented study is to provide a theoretical framework, methodological insights and experimental design to model relevant metacognitive processes, and specify a set of recognizable and measurable indicators to assess metacognition in dialogue. the paper is structured as follows. section 2 reviews methods to assess metacognition in interactive setting. in section 3, we specify the model of metacognitive processes within the framework of dynamic interpretation theory (dit). we adapt the established metacognition assessment instruments in order to discover potential correlations between dialogue acts and metacognitive events. section 4 presents experimental design featuring data collection, processing and iso 24617-2 compliant annotation protocols. we wrap up the paper by outlining expected project outcomes. ## metacognition assessment instruments assessment of metacognition traditionally involves self-reported measurements. the most widely used metacognition questionnaire @xcite ) evaluates fac-tors related to positive and negative metacognitive beliefs, metacognitive monitoring and judgements of cognitive confidence. questionnaires are however of limited value since they are subjective and not always accurate @xcite . there are two online methods proposed to assess metacognition: thinking aloud and reflection when prompting. participants speak about their own cognitive states or processes and their understanding of partner's states and processes, or are prompted to reflect on the reasons why they chose specific actions -verbalized metacognition. the methods enable assessment of three elements of metacognition -experiences (e.g. confidence, confusion), knowledge (e.g. gaps), and strategies (e.g. actions). think-aloud and prompting protocols provide rich information about the metacognitive processes when performing a task and are powerful predictors of test performance @xcite . verbalization methods are proven valid, but time consuming. moreover, elicitation of explicit monitoring, reflection and regulation moments may disrupt or even break down the interaction process, distort its naturalness, trigger attention theft, increase cognitive load and impact negatively participants' engagement. there is research performed on the psychophysiological measurement of metacognition. physiological measures make use of eeg electroencephalography (wokke et al., 2020), heart rate @xcite , and pupil dilation @xcite , but require rather complex and often expensive hard-and software set ups. other methods exploit information about interlocutor's behaviour via log files and efficiently combine it with questionnaire data @xcite . recently, increasing computational power and technological advances opened up new data-driven assessment scenarios. a huge diversity of inexpensive tracking and sensing devices enable rather exhaustive real-time monitoring and immediate assessment of affective cognitive states, including metacognitive aspects @xcite . significant progress has been booked in automatic affective cognitive state recognition from speech and visual signals @xcite . large amounts of multimodal data is used to train deep learning algorithms to recognize facial expressions related to emotions and cognitive states in large variety of scenarios. the definition and detection of metacognitive multimodal indicators requires transforming the raw multi-sensory data collected in a meaningful way so that it allows taking decisions, provide indicators of interlocutor's performance, efficiency and preferences @xcite . this has been done for interaction logs, the records of sequential actions users performed in an interface. such actions are interpreted as any communicative action, i.e. having certain communicative functions. a set of dialogue acts has been proposed for screen events by translating the human-human communication mechanisms into human-computer interactions as functions of @xcite . coherence and interaction analysis is applied to analyse think-aloud interviews and prompting interaction transcripts @xcite ; modern natural language processing techniques are used @xcite . in multimodal interactions that involve speech, taking notes, nonverbal communication and graphical user interface actions, metacognitive strategies are observable via interaction logs, metacognitive experiences -via recorded and tracked behaviour, and metacognitive knowledge -via speech and typed transcripts. the interaction-based approach to measure metacognition that we propose will enable real-time and non-intrusive assessment of all metacognitive aspects -experiences, knowledge and strategies. ## modelling metacognitive processes in dialogue interaction metacognitive regulation refers to adjustments individuals make to their processes to help control their task performance, learning and interaction. metacognitive processes underlie awareness, monitoring, reflection and regulation activities @xcite . metacognition has implicit and explicit forms, foot_0 and is applied to own (sender's) and others (addressee's) cognitive processes. in human dialogue, metacognitive processes concern reasoning about interlocutors' intentions and knowledge, and are often modelled as parts of shared or mutual beliefs forming a common ground @xcite . common ground is not directly accessible. an access to self and others cognitive processes through questionnaires and think-aloud protocols is very limited; reports on own and others' intentions can be inaccurate. (meta)cognitive processes underlying establishing and updating common ground (grounding), on the other hand, may become accessible through or inferred from observable dialogue behaviour. for instance, gaze (re-)direction deliver information about the interlocutor attention by means of frequency and duration of gaze fixation on the areas of interest (aoi), but also provides an evidence about the positive versus negative emotional reaction on the fixated object. in face-to-face conversation, participants may present evidence of grounding through verbal and vocal signals, body movements and facial expressions; in interaction with graphical user interfaces, typing behaviour, mouse movements and clicks may signal changes in (meta)cognitive and motivational functioning. a metacognitive event is characterised through evidence of reflexive activities indicating any level of sender's mindful awareness about own (sender's) and others (partner's) cognitive process(-es): • level 0: ignore or offer false continuation; • level 1: pay and secure attention (mutual eye contact); • level 2: recognise, record change and respond with minimal signals (gaze (re-)direction, head nods, 'mmhmm', 'uhu'), check out and verify recognition; • level 3: interpret, check out and verify understanding, and respond to content and feeling ('i see what your mean...', 'i am confused...'); • level 4: evaluate content and feeling, inspect/compare past experiences and verify hypotheses ('i am as worried as you are...'); • level 5: regulate and align, correct/adjust, imitate, anticipate consequences, plan the ongoing procedure (content, sequences, timing,...). at all these levels, positive and negative beliefs concern sender's awareness about: (i) his/her own thoughts (zero-order theory of mind abilities, premack and woodruff (1978)), (ii) about another person's thoughts (first-order theory of mind), and (iii) what another person thinks about sender's thoughts (second-order theory of mind). consider the following example 2 : (1) du1. a: the next train is at 11:02. du2. b: at 11:02. du3. a: that's correct. du4. b: okay thanks in 1, a in order to continue the dialogue should know that b understands his utterance du1 and believes its content p. b's utterance du2 can be 2 adapted from @xcite . at this point a does not yet know whether his utterance has reached b and was well understood. b's next contribution du4 provides evidence for that; upon understanding du4, a has accumulated the following beliefs: (2) a believes that p a believes that b believes that p a believes that b believes that a believes that p a believes that b believes that a believes that b believes that p a believes that b believes that a believes that b believes that a believes that p or represented as mutual beliefs equal to: to classify and model implicit and explicit metacognitive events (acts), the framework of the dynamic interpretation theory (dit, @xcite ) and the iso 24617-2 dialogue act annotation standard (iso, 2012) will be used. dit has emerged from the study of multimodal human-human dialogues uncovering fundamental principles observed in such interactions. dit and its subset iso 24617-2 are open multidimensional dialogue act taxonomies foot_2 . they are proven to provide theoretically grounded and empirically tested inventory of dialogue acts with fine-grained semantic distinctions presenting the semantic framework for the systematic analysis and computational modelling of multimodal dialogue behaviour in many interactive settings. special attention will be paid to feedback acts which we assume are crucial for successful recognition of metacognitive events: positive and negative feedback about speaker's own (auto-feedback) and the partner's processing (allo-feedback) at the five processing levels: attention, perception, interpretation, evaluation and execution @xcite . speaker's repairs, (self-)corrections, partner completions and hesitations (silent and filled pauses) are assumed to strongly correlate with moments of reflection and may reveal speaker's cognitive confidence. managing allocation of time, turn, structuring discourse and control over issues under discussion concern with planning aspects. analysing socio-emotional aspects will enable modelling of metacognitive activities related to positive, negative thoughts, feelings of uncontrollability and danger, and engagement related emotions such as boredom, enjoyment and frustration. ## experimental design use case the importance of metacognition has been empirically proven for negotiations @xcite . high self-and others-monitors are more concerned that their negotiations go well, flexibly modify their actions to better adapt to the changing dynamics of the situation, typically by using other people's behaviour as a guide to their own. high self-monitors and -assessors are more likely to engage in argumentation and are better able to accomplish their goals. as the use case, we will focus on patientphysician negotiations for shared decisions. medi-cal students and professionals tend to overestimate the value of medical knowledge and are known as poor self-monitors and self-assessors @xcite . therapy planning scenarios of varied complexity will be defined reflecting different participant's dispositions. interaction concerns multiissue bargaining where each issue involves multiple negotiation options with preferences representing parties negotiation positions. preferences are weighted in order of importance (strength) and defined as the participant's beliefs about attitudes towards certain behaviour and abilities to perform this behaviour. the goal of each partner is to find out preferences of each other and to search for the best possible mutual agreement. the human participant -doctor -negotiates either with a human or artificial patient who will have different preferences and instructed (programmed) to apply several negotiation and decision-making strategies @xcite . 4 data collection will be performed via a) human-human role-playing (small-scaled) and b) human-agent interactive simulations (large collections). role-playing method is often used to collect interactive data in a controlled setting and underpins simulations of many real-life communicative situations @xcite . here, one participant will be randomly assigned the role of a doctor, the other participant -a patient. each participant will receive instructions and preference profile, and asked to negotiate a mutual agreement with the highest possible value. procedures will be specified for the settings where both participants: (1) observe others' actions and flag problems or gaps; simulations of communicative situations with human and artificial simulated patients (sps) will be arranged. regular medical communication practice often takes place in a patient-simulated setting, where simulated patients (sps) are involved to portray a particular set of symptoms or roles @xcite . simulations with humans provide high fidelity training, but are costly, difficult to reproduce and access. ai agents as sps can be used to create specific situations in which physicians metacognitive processes can be activated and assessed @xcite . moreover, simulations will impose certain restrictions in order to investigate a controlled set of communicative (metacognitive) activities and related phenomena without having to deal with unrelated details. multimodal data will be recorded. the quality of recordings will be adapted to the application conditions, i.e. a fairly good but not perfect acoustic and visual quality will be targeted. prior to recordings, participants will complete the short mcq-30 questionnaire. we will account for gender and role differences. ## expected outcomes the proposed project will contribute to a better understanding of metacognitive processes underlying dialogue participants decision-making and interactive performance progressing towards a computational cognitive model of social metacognition. an interaction-based method for metacognition assessment will be worked out providing an isocompliant typology of metacognitive events, a set of multimodal feature extraction and classification models as well as new tools for multidimensional dialogue analysis. finally, substantial amount of multimodal data annotated with the iso 24617-2 dialogue acts will be provided to the research community via dialogbank release. 9