Dual Process Masking for Dialogue Act Recognition (2024)

Abstract: Dialogue act recognition is the task of classifying conversational utterances based on their communicative intent or function. To address this problem, we propose a novel two-phase processing approach called Dual-Process Masking. This approach streamlines the task by masking less important tokens in the input, identified through retrospective analysis of their estimated contribution during training. It enhances interpretability by using the masks applied during classification learning. Dual-Process Masking significantly improves performance over strong baselines for dialogue act recognition on a collaborative problem-solving dataset and three public dialogue benchmarks.

## introduction dialogue act recognition is the task of classifying utterances in a conversation based on their communicative intent @xcite . accurately discerning the intent of each utterance in humanto-human or human-to-machine interactions plays a central role in a broad range of applications such as conversational agents @xcite , meeting analysis @xcite , and emotion analysis @xcite . however, defining a universal dialogue act taxonomy is challenging because conversational settings vary widely. different domains use distinct taxonomies that reflect their specific purposes and characteristics. hence, automatically capturing the varied linguistic structures associated with dialogue acts requires a large volume of labeled training data that encompasses a wide range of permutations. however, the availability of labeled data in many domains is limited because the labeling process is labor-intensive. there is growing demand for focused analysis of small-scale conversational data across specific domains (e.g., education, science, healthcare, transportation), where specialized taxonomies are often employed @xcite . despite advances in dialogue act recognition via deep learning @xcite @xcite and large language models (llms) @xcite @xcite , the effects of limited data and specific tokens on performance remain understudied. in particular, existing approaches based on deep learning, including llms, achieve high recognition rates on large-scale data, but their low interpretability limits their applicability to other domains and different dialogue act taxonomies. to bridge this gap, we propose dual-process masking (dp-masking), which uses masking to de-emphasize tokens that are less relevant to dialogue act recognition and uncover the functional language structure indicative of dialogue acts. as shown in figure 1 , dp-masking reduces noise by identifying essential structures, thereby enhancing dialogue act recognition accuracy in datasets with limited data. it also improves interpretability by elucidating key tokens crucial for different dialogue acts. dp-masking draws from dual-process theories in cognitive psychology @xcite @xcite , which describe two systems of human cognitive processing. the intuitive system is fast, automatic, and instinctive, relying on heuristics and past experiences, while the reflective system is slow, controlled, and rational, requiring deliberate effort for complex problem-solving @xcite . these two systems operate in parallel and influence human intelligence and decision-making. in dp-masking, the reflective system iteratively masks less relevant tokens by learning the tokens' contributions to dialogue acts through controlled effort, while the intuitive system internalizes this learning for rapid, automatic classification. we validate dp-masking on a collaborative problem-solving (cps) dataset collected from student interactions with a collaborative game-based learning environment, as well as with three largescale public dialogue corpora. this paper primarily focuses on the cps data because of its limited training set size (∼2k labeled instances). the other datasets are employed to examine dp-masking performance across varying dataset sizes. for the cps dataset, dp-masking achieves state-of-the-art (sota) results. we also experiment with different implementations of dp-masking using llama-2-7b, llama-3-8b @xcite , and flan-t5-small @xcite as the underlying model and show how dp-masking outperforms random masking and a no-masking approach, with an average normalized improvement of 13.4% and 17.6% in macro f1 scores, respectively. our analysis shows how certain token characteristics are associated with masking performance as an interpretation of the learned model and how dp-masking could lead to performance improvements for dialogue act recognition, highlighting the implications of this research. for the three public benchmarks, dp-masking achieves significant improvements with limited data, but its benefit decreases as the data size increases. with the full dataset size, dp-masking shows performance comparable to the underlying models, while achieving sota performance on one of the three benchmarks. overall, dp-masking outperforms random masking with an average normalized improvement of 3.2% and a no-masking baseline strategy with an 8.3% improvement in matthews correlation coefficient (mcc). the contributions of this work include: (1) a novel dialogue act recognition framework, dp-masking, inspired by dual-process theories of cognition; (2) an evaluation of dp-masking on a limited-sized educational dataset and three large-scale public benchmarks; (3) an analysis of recognition results to assess their masking strategies. ## problem definition given an input dialogue, u , consisting of a sequence of utterances, @xmath0, u 2 , ..., u j ), our goal is to output a dialogue act label, l @xmath1, for every utterance u i . each utterance is represented as a sequence of tokens: u @xmath2, w 2 , ..., w k ) where @xmath3. the input in our experiments consists of the target utterance, u i , and k preceding utterances (u i-k , ..., u i-1 ) as context. however, for simplicity, we refer to the concatenation of context and target utterance as an utterance. ## dual-process masking inspired by dual-process theories of cognition, we implement a dual-process masking learning algo-rithm where the intuitive learning stage and the reflective learning stage iteratively interact and influence dialogue act recognition. figure 2 shows the overall workflow of dp-masking. the reflective learning stage actively identifies mask tokens that lead to correct dialogue act labels, and the intuitive learning stage consolidates this knowledge into long-term memory for classification. the core components are a masker m and classifier c pair for each of the reflective learning r and intuitive learning i stages, and a working memory, w , connecting these two stages. in the reflective learning stage, target masks, which a masker aims to output, are initialized with random masks in the utterances, and this stage's masker m r learns these target masks using the training data of unmasked utterances. then, the reflective classifier c r learns to classify utterances masked by m r into dialogue act labels. the classification results are analyzed, and the successful masks (i.e., masks from the correctly classified utterances) are updated in the working memory w . in the next iteration, the target masks are replaced with successful masks and supplemented with random masks for exploration. in the intuitive learning stage, the intuitive masker m i learns only from successful masks stored in w , and the intuitive classifier c i learns classification using utterances masked by m i . algorithm 1 outlines the learning algorithm for dp-masking. here, we employ a pretrained language model as the underlying model for each masker and classifier. first, w is initialized by copying training data, u tr (line 1). the algorithm goes through multiple epochs of reflective learning and intuitive learning stages (line 2). in the reflective learning stage, we define three key steps: exploration, analysis, and information acquisition. for exploration, p% of the tokens in u tr (stored in w ) are masked randomly and then u tr is randomly split into reflective sub-training (u r_tr ) and ## collaborative problem-solving data in this section, we conduct an experiment using a collaborative problem-solving (cps) dataset, which exemplifies our target limited data setting. data. the cps dataset consists of dialogues between groups of three to four middle school students collaboratively working in an educational game, ecojourneys, as part of their science class on ecosystems to solve a mystery centered on an illness spreading among fish in local farms on a fictional island @xcite . students communicated via an in-game chat interface throughout the game sessions. to annotate this data, educational researchers designed the dialogue act taxonomy based on four key components of cps behavior: sharing ideas, negotiating ideas, regulating problem-solving activities, and maintaining communication @xcite . we expanded this taxonomy by adding 'other' and 'off-task' categories to more comprehensively represent the dialogue acts observed in the chat during middle-grade collaborative problem solving. two senior graduate students with expertise in cps performed the annotation under the supervision of a senior professor in educational research. initial annotation of 20% of the data demonstrated a high inter-rater reliability (irr) with a cohen's kappa of 0.81 @xcite , indicating strong agreement @xcite . the two annotators then discussed and resolved their disagreements. afterward, one of the annotators completed annotating the rest of the dataset, following the guideline for coding qualitative data @xcite . after removing non-dialogue messages, the dataset consists of 1,990 utterances. the data collection was conducted with institutional review board (irb) approval, and the anonymized cps data is available for research purposes upon request. compared methods. the baselines include bert-base models (110m parameters), including bert, bert-lstm, and bert-bilstm, along with three dialogue act recognition approaches, sgnn @xcite , casa @xcite , roberta-base (125m) and roberta-large (355m) @xcite , which are publicly available and are state-of-the-art models. 2 we also compare dp-masking (dp-*) with random masking (r-*) and no-masking. we apply these three masking strategies to three underlying models: flan-t5-small (80m), llama-2-7b, and llama-3-8b. to improve training efficiency given the high fine-tuning costs of llama models, only the classifier was trained on llama, using masked inputs from the ft5 masker. all llama models were fine-tuned using qlora (8-bit quantization) @xcite . hyperparameters are described in appendix c. ## analysis we analyze learned masking strategies focusing on 1) masking rate, 2) mask distribution by part of speech, 3) mask types, 4) effect of input length, 5) accuracy by class, and 6) qualitative analysis with successful and failed masking examples. masking rate. the masking rate is defined as the fraction of tokens masked in each dataset. for dp-ft5, the masking rates were 13.5% on the cps dataset (2k) and 6.8% on the larger talkmoves datasets (170k), described in section 7. the lower masking rate in talkmoves suggests that the classifier, which was trained on extensive data, performs well during the iterative training process before the masker has fully learned masking. this variability in masking rates, tailored to the characteristics of the data, contrasts with the fixed 15% masking rate of bert style models. ## experiment: public benchmark we also evaluate dp-masking on three large-scale public datasets and analyze the impact of data size on the effectiveness of masking approaches. data. the oasis dataset @xcite contains 1219 transcripts of live calls made to british telecommunications and operator services with 42 dialog acts. we adopted the same data split as used in @xcite . the talkmoves dataset @xcite consists of 501 written transcripts of conversations between teachers and students in kindergarten through 12th grade math lessons with 6 talk move (dialogue act) labels. switchboard dialog act (swda) @xcite is a telephone speech dataset with about 2,400 two-sided conversations among 543 speakers with 42 dialogue acts @xcite . see compared methods. we chose flan-t5-small (ft5) as the underlying model because it is lightweight (low training and inference costs) and performs competitively on the cps data. we compare the following masking approaches. (1) ft5: ft5 with no masking. (2) r-ft5: random masking with ft5, following roberta @xcite . (3) dp-ft5: dual-process masking with ft5 (our proposed method). hyperparameters are described in appendix c. ## conclusion we have introduced the dual-process masking (dp-masking) framework for dialogue act recognition. dp-masking consists of a reflective learning stage for extracting successful masks and an intuitive learning stage for encoding refined information into long-term memory. the results, validated across collaborative problem-solving dialogue data and three public dialogue benchmarks, reveal three key findings: (1) dp-masking achieved improvements over competitive baseline models in challenging scenarios with limited data. (2) it demonstrated robust performance on public benchmarks, establishing a new sota for a benchmark task. (3) it enhanced interpretability by assessing the contributions of masked tokens to dialogue act recognition. future work can explore incorporating various reflection techniques to identify performance improvement points, integrating feedback on this work's findings into the learning process, and investigating whether dp-masking enhances performance in other natural language processing tasks such as question answering, summarization, and document classification. the implementation of our work is available at https://github.com/ykim32/dpmasking .