Improving Multi-label Malevolence Detection in Dialogues through Multi-faceted Label Correlation Enhancement (2022)

Abstract: A dialogue response is malevolent if it is grounded in negative emotions, inappropriate behavior, or an unethical value basis in terms of content and dialogue acts. The detection of malevolent dialogue responses is attracting growing interest. Current research on detecting dialogue malevolence has limitations in terms of datasets and methods. First, available dialogue datasets related to malevolence are labeled with a single category, but in practice assigning a single category to each utterance may not be appropriate as some malevolent utterances belong to multiple labels. Second, current methods for detecting dialogue malevolence neglect label correlation. Therefore, we propose the task of multi-label dialogue malevolence detection and crowdsource a multi-label dataset, multi-label dialogue malevolence detection (MDMD) for evaluation. We also propose a multi-label malevolence detection model, multi-faceted label correlation enhanced CRF (MCRF), with two label correlation mechanisms, label correlation in taxonomy (LCT) and label correlation in context (LCC). Experiments on MDMD show that our method outperforms the best performing baseline by a large margin, i.e., 16.1%, 11.9%, 12.0%, and 6.1% on precision, recall, F1, and Jaccard score, respectively.

## introduction safety is an increasingly important aspect of artificial intelligence development @xcite @xcite . when it comes to dialogue agents, taking measures to avoid risks of generating undesirable and harmful responses may have a profound positive impact on the adoption of conversational technology @xcite . research on safe dialogue agents involves aspects such as inaccurate information @xcite , fairness @xcite , and unauthorized expertise @xcite . malevolence is another key aspect @xcite , e.g., whether the dialogue utterance contains malevolent content that is related to offensiveness @xcite , toxicity @xcite , ad hominem @xcite , and toxicity agreement @xcite , etc. there have been several studies targeting malevolence detection @xcite @xcite . we build on the work of @xcite who introduce the malevolent dialogue response detection and classification task, present a hierarchical malevolent dialogue taxonomy, create a labeled multi-turn dialogue data set, and apply state-ofthe-art text classification methods to the task. one important limitation of their work is that they only explore single-label dialogue malevolence detection (sdmd), i.e., they assume that each dialogue utterance corresponds to a single malevolence or non-malevolence label. however, some utterances have more than one label, e.g., in figure 1 , the utterance "f** people are disgusting" foot_0 belongs to both "disgust" and "negative intergroup attitude @xcite ." this is because malevolence labels are correlated with one another, which we refer to as label correlation in taxonomy (lct). @xcite propose a hierarchical malevolent dialogue taxonomy that classifies correlated malevolence labels into the same group by investigating three dimensions -negative emotions, negative psychological behavior, and unethical issues. however, the correlation of malevolence labels in different groups is not well captured. another limitation is that the above studies neglect the impact of malevolence in dialogue contexts (i.e., previous turns) on the current utterance. previous work concatenates the dialogue context as model input without explicitly modeling the malevolence figure 1 : label correlation in taxonomy (lct) and label correlation in context (lcc). in terms of lct, "negative intergroup attitude (nia)" is correlated with "disgust", which can be reflected by the utterance in blue (lct). in different turns, "blame" is likely to cooccur with "anger" and "blame", which can be reflected by the utterances in green (lcc). transition. for example, in figure 1 , "blame" is likely to cause "blame" for the same person, while for different persons, "blame" is likely to cause "anger". this is due to label correlation in context (lcc). zhang et al. (2021b) do not take correlation of malevolence labels in different dialogue turns into account and our label-correlation mechanisms are different from previous methods which require multi-label training sets @xcite . we address the two limitations listed above. our goal is to boost multi-label dialogue malevolence detection (mdmd) by incorporating label correlation in taxonomy and context based on a singlelabel dataset with re-annotated multi-label evaluation data. this goal comes with two main challenges: (1) a dataset challenge, as we only have one label per utterance in the training data, which increases the negative effect of unobserved labels during training: how to improve the single gold labels via lct and decrease the probability of overfitting; (2) a classification method challenge: how to capture lcc to help improve the classification. based on conditional random field (crf), we propose a multi-faceted label correlation enhanced crf (mcrf) framework to improve mdmd from single-label training data. the approach contains a position-based label correlation in taxonomy (plct)-based encoder and a multi-faceted crf layer, which includes a lcc-based feature function and lct-based label distribution learning. for the dataset challenge, we build a lct-based label distribution learning module to exploit the label correlation in hierarchical taxonomy, which can alleviate the unobserved label problem. for the classification method challenge, we build an lccbased transition function to exploit the label correlation in context. we crowdsource a new dataset based on the previously released malevolent dialogue response detection and classifying (mdrdc) dataset, conduct experiments on this dataset, and show that mcrf with a pretrained model, i.e., bert-mcrf, outperforms competitive baselines by a large margin. we also conduct further analyses of the lct and lcc modules, which reveal that multi-faceted label correlation does enhance multi-label dialogue malevolence detection. we summarize our contributions as follows: (1) we crowdsource a new dataset, i.e., mdmd, for the task of multi-label dialogue malevolence detection from single-label training data. ( 2 ) we propose multi-faceted label correlation, including lcc and lct, which is shown to be beneficial for dialogue malevolence detection. (3) we introduce a new framework, mcrf, and compare it with competitive baseline models on the mdmd dataset and demonstrate its effectiveness. ## experimental setup we conduct experiments to answer the following research questions: (rq1) how does bert-mcrf compare to baselines on the mdmd test set? (rq2) what is the impact of the number of labels on the performance of bert-mcrf? (rq3) what is the influence of different lct and lcc settings? (rq4) what do the components of bert-mcrf contribute to its overall performance? ## conclusion and future work we have studied multi-label dialogue malevolence detection and built a dataset mdmd. the dataset statistics suggest that the dataset quality is substantial and that it is essential to do multi-label dialogue malevolence detection as almost 12% of the utterances have more than one malevolent label. we have proposed bert-mcrf by considering label correlation in taxonomy (lct) and label correlation in context (lcc). experimental results suggest that bert-mcrf outperforms competitive baselines. further analyses have demonstrated the effectiveness of lct and lcc. a limitation of bert-mcrf is that it is not good at detecting implicitly malevolent utterances, e.g., "deceit." as to future work, we plan to address this type of utterance and investigate how to enhance bert-mcrf in terms of implicit multi-label dialogue malevolence detection by semi-supervised learning as there are large-scale unlabeled datasets.