BPM _ MT : Enhanced Backchannel Prediction Model using Multi-Task Learning (2021)

Abstract: Backchannel (BC), a short reaction signal of a listener to a speaker’s utterances, helps to improve the quality of the conversation. Several studies have been conducted to predict BC in conversation; however, the utilization of advanced natural language processing techniques using lexical information presented in the utterances of a speaker has been less considered. To address this limitation, we present a BC prediction model called BPM_MT (Backchannel prediction model with multitask learning), which utilizes KoBERT, a pre-trained language model. The BPM_MT simultaneously carries out two tasks at learning: 1) BC category prediction using acoustic and lexical features, and 2) sentiment score prediction based on sentiment cues. BPM_MT exhibited 14.24% performance improvement compared to the existing baseline in the four BC categories: continuer, understanding, empathic response, and No BC. In particular, for empathic response category, a performance improvement of 17.14% was achieved.

## introduction backchannel (bc) is a short and quick reaction, such as "uh-huh" and "yes", of a listener to speaker's utterances in a conversation @xcite . timely and appropriate use of bc in a conversation is important because bc has various functional categories @xcite ; and each category has different roles and influences on enriching a conversation @xcite @xcite . it is important to understand the speaker utterances for the proper use of bc. in this paper, we introduce a method to enhance the utilization of lexical information in utterances for bc category prediction. utterances can be expressed by the acoustic and lexical information in spoken dialogue. in early bc prediction studies, there were several cases in which only the acoustic features of utterance were used @xcite @xcite . recently, the additional use of lexical information has improved the performance of the model @xcite . however, the use of such lexical information was rather simple, including part-of-speech of the last word of utterance @xcite or embedding features at the word level, such as word2vec. @xcite used word2vec and reported the state-ofthe-art performance level of about 58% accuracy on three types of bc categories, i.e. continuer, assessment, and no bc. @xcite also selected an approach similar to that of @xcite . however, they focused on addressing the latency of bc responses, not on utilization of lexical information. in this study, we introduce a bc prediction model using multi-task learning (bpm_mt), which learns the main task of bc category prediction and the sub-task of predicting sentiment score simultaneously. the four contributions of this paper are as follows. (1) our model predicts a novel bc category of "empathic response", which has been identified in existing research, but has not been previously considered. as the results of the experiment, bpm_mt showed a performance improvement of 14.24% compared to the baseline model @xcite , and in particular, empathic response prediction improved by 17.14%. overall, we demonstrated the effectiveness of enhanced utilization of lexical information on bc prediction. ## backchannel prediction model the main task of the model is to take a patient's utterance as input and predict a category of bc generated by a doctor. according to the presence of a sub-task that learns the sentiment score of the utterance, two types of models were designed for the performance experiment. one model was the single task model (bpm_st) that included only the main task. the other model was the multi-task model (bpm_mt) that included both the main and sub-tasks. both models used audio and text as input data. figure 2 illustrates the models. ## results and discussion the experimental results of four categories are listed in regarding the context length, there was an improvement in the performance of up to a minimum of 5.78% (20 words) and a maximum of 14.24% (5 words) compared to the baseline model of ortega. the bpm_mt model outperforms the baseline models even when the input context is limited to 5 words. therefore, the bpm_mt can be used when less context information is available with a small number of contextual word data is available as input. among the bc categories, the performance improvement for the empathic response category was the greatest in comparison with the baseline models. when the context length was 10, the performance increased by 17.14% compared to the or-tega model. this emphasizes the importance of understanding the speaker's sentiment in utterances or the flow of conversation to express the signal of empathy. compared to bpm_st, bpm_mt showed the highest improvement (5.6%) for the continuer category. figure 3 : distribution of positive and negative sentiment words by bc category in the context length equals to 5 (*@xmath0.05). figure 3 shows the distribution of positive and negative sentiment words depending on bc categories. we used the data of the context length equals to 5, because setting the model with that length showed the highest performance improvement when the sub-task was applied. to compare the results by bc category, we ran anova with the tukey's posthoc test. the result shows significant differences of the means among categories (f(3, 20549) = 2013.7, @xmath1.05 for the positive sentiment; f(3, 20549) = 5318.6, @xmath2.05 for the negative sentiment). more cases of significant differences were shown in the negative sentiment. this indicates the importance of negative sentiment information that exists in patient utterances in bc category prediction. when comparing the categories of no bc and bcs, richer sentiment information induced bc generation in both positive and negative sentiments. these indicate that additional learning about sub-tasks was an appropriate approach to improving bc performance. ## conclusions we presented a bc category prediction model that achieved a performance of 76.69% by utilizing lexical information with sentiment cues. across the three types of bc categories, the empathic response category, which hasn't been previously measured, showed a performance improvement of 17.14% compared to the model of ortega. this increase implies that utilization of lexical information helped in predicting empathy function. the limitation of this study is the imbalance in the number of data according to the bc category. because this may be a characteristic of the counseling data used in the study, we may have to check other types of bc data. in the future, we plan to further improve the performance of the bc prediction model using the attention mechanism between different modality information @xcite by simultaneously receiving multi-modality information as input.