Two-level classification for dialogue act recognition in task-oriented dialogues (2020)

Abstract: Dialogue act classification becomes a complex task when dealing with fine-grain labels. Many applications require such level of labelling, typically automatic dialogue systems. We present in this paper a 2-level classification technique, distinguishing between generic and specific dialogue acts (DA). This approach makes it possible to benefit from the very good accuracy of generic DA classification at the first level and proposes an efficient approach for specific DA, based on high-level linguistic features. Our results show the interest of involving such features into the classifiers, outperforming all other feature sets, in particular those classically used in DA classification.

## introduction task-oriented dialogue systems are specific in many respects. first, and this is the most important characteristic, they focus on a limited semantic domain, rendering the comprehension operation closer to slot filling than deep semantic understanding. moreover, these applications correspond to very specific interactions, usually with one of the speakers (the human or the machine) leading the dialogue. in such situations, recognizing dialogue acts (da) becomes extremely useful as a preliminary step of the understanding process: associating speaker's utterances to das makes it possible to identify very efficiently the type of information they bear and the knowledge to be conveyed. many works on dialogue act classification have been done for a long time @xcite . the first question consists in defining the set of relevant dialogue acts for the system. different generic da annotation schemes have been proposed, including an iso standard @xcite . in addition, several dialogue corpora have been annotated and made available, among which two are particularly used: switchboard dialog act corpus, (swda) @xcite and meeting recorder dialog act (mrda) @xcite . these datasets have allowed to train different classifiers. in these works, two preliminary questions arise: the identification of the dialogue acts to be classified and the features on which the classifier must be based. in most cases, the classification task targets a limited set of classes (corresponding to very general das such as statement, question etc.) and features remain at a low level (n-grams of words, characters, word embeddings, etc.). the performance of these classifiers is generally very good. however, the proposed task (i.e. the targeted tagset) remains often too general for an optimal use in a dialog system. moreover, and this is a recurring problem in this type of approach, it is sometimes difficult to interpret the results and understand precisely the relative impact of the different features on the model. in this paper, we address the question of da classification for helping comprehension and dialogue supervision in the context of task-oriented dialogue systems. several works have shown the importance of das for guiding dialogues, in particular in the context of adaptive and socially-aware systems used for training social skills @xcite . in such cases, the dialogue system plays the role of a person to whom information must be transmitted @xcite . the targeted information, the semantic context, as well as the way the information should be transmitted are known in advance by the system. we are then in the situation where the most difficult task for the system is to generate an appropriate reaction to the way the human has delivered the information: appropriate feedbacks, clarification questions, surprise, emotional reactions, etc. this is the reason why da classification plays an important role, more than a classical understanding or intent/slot-filling @xcite . the identification of dialogue acts can therefore be an extremely efficient pre-processing for comprehension, provided that das are specific enough to guide the system efficiently. da identification comes to a classification task which requires three preliminary steps: identifying an appropriate tagset, collecting an adequate corpus and annotating it. the annotation stage is an important issue, this task being done mainly manually. as a consequence, the corresponding datasets remain rather small, which can be problematic for applying machine learning techniques. moreover, as described in this paper, an efficient da classification makes use of a large number of features. one way to address this question of high dimensionality is to study precisely the features to be involved in the model, their characteristics and behavior. another characteristics of dialogue systems is that comprehension has to be done incrementally and in real time. when modeling a dialogue based on natural dialogue corpora, the global set of speakers' utterances are taken into account, making it possible to analyze large sequences. however, for automatic dialogue systems, the only utterances containing information to be processed are those already produced by the human speaker. this has important consequences on the technique to be applied: it cannot take into account a dialogue in its entirety and then cannot be bi-directional (no possible forward looking when calculating the system's answer): the model should mainly/only rely on the user's utterances (utterances produced by the system being limited to speaker's reactions). we show in this study that da classification for these specific dialogue situation can be based only on user's utterances, offering the possibility of an online and real time processing. this paper proposes a general methodology to address the problem of classifying specific dialogue acts (in place of generic das) based on a set of interpretable features, adapted to the characteristics of data available for this type of task. we propose in a first part a summary of the existing techniques and current results in da classification. in the second section, we describe the specific data we are working on and the constitution of the dataset. the rest of the article is devoted to the presentation and discussion of the results. ## dialog act classification dialogue acts (das) correspond to communicative functions describing the illocutionary force (speaker's intention) that can be associated with utterances in the discourse. da classification consists in associating to each segment of the discourse (that can be a turn, an inter-pausal unit or a segment returned by a speech recognition system) a label corresponding to the communicative function. da classification is a classical problem for which many propositions have been done @xcite @xcite @xcite @xcite . two preliminary questions must be addressed in this task before a classification technique can be applied: the choice of the tagset, and the identification of the features used by the classifiers. we propose in this section to quickly address these issues before presenting the main results available in the literature. ## the dataset the goal of this paper is to design a da classification method providing inputs for the understanding module of a dialogue system. we give first details about the dataset and the use case before describing the annotation process. ## feature selection we propose to explore different approaches for classifying da in the context of our specific dataset. more precisely, we will work specifically on two aspects: the feature set (by adding high-level features on top of the classical ones) and classification techniques (by comparing a classical multiclass approach with a specific case of hierarchical classification). as it is usually the case, a preliminary step of data cleaning is applied, suppressing special characters, accents, stopwords and normalizing the font case. a separate pre-processing has been done, adding lemmatisation to these operations. moreover, we used marsatag @xcite , a lexical processing tool integrating different functionalities (including pos tagging) in order to extract the lexical features. classical features : we first experiment classification with a set of features classically used for da identification. it consists in combining tf-idf principles with word and character n-grams. applying a principal component analysis, we extracted 4 combinations to be tested: morpho-syntactic features : we propose to explore the role of low-level morpho-syntactic features, based on pos categories: ## flat classification our choice of the classification techniques takes into account two aspects: the size of the dataset and our intention to have interpretable models. these two goals can be contradictory. having a rather small dataset increases the risk of overfitting, which leads to chose the simplest models. on the other hand, in order to improve interpretability, we want to integrate as many linguistic features as possible. we chose in this perspective to test different simple machine learning models: logistic regression, support vector machine, k-nearest neighbors, decision tree, random forest; some of them are known to be particularly adapted to learn on small dataset @xcite . different regularization techniques has been applied according to the model: l1, l2, elasticnet, as well as tuning the margin parameter for svm or the tree depth for random forests. feature selection is then applied using an anova to select the k-best features. the best classifier is retrained with this new feature set. two classical methods for hyperparameter tuning, randomsearch and gridsearch, have been applied on the training set, with k-cross validation. considering the size of the dataset, a k-fold cross-validation has been applied, with @xmath0, in order to have a validation set large enough to be statistically significant (different values of k have been tested). moreover, as shown in the previous section, class distribution is imbalanced, some of them representing 0.3% of the data, where others correspond to 20%. we used synthetic minority oversampling technique (smote) which consists in adding new points by combining the points of the minority class with the closest neighbors. we refined this method by filtering the output with tomek link removal, removing the points that are the closest neighbors of each other. as a result, different tests have been done on imbalanced and balanced classes (with the same class dm number of discourse markers in the utterance fp number of filled-pauses tok number of tokens in this first experiment, we classified directly, with a unique classifier, the 13 classes: init state, init remediation, bad new, bad new remediation, current remediation, current state, opening, closing, discourse, explication, question, reassurance, social interaction. several tests have been done with different feature combinations (note that clit i stands for clitic 1st to 3rd person): ## two-level classification multiclass classification is always prone to several difficulties. moreover, the identification of classes corresponding to fine-grained knowledge (e.g. state, remediation, etc.) used in the perspective of dialogue understanding is much more difficult than other types of classification. addressing this issue can be done thanks to hierarchical classification @xcite . chaining classifiers in particular when there is a lot of classes to consider can be extremely helpful by breaking down the problem into small simpler problems, increasing the overall performance. as explained above, at the difference with da classification triggering meta-classes, we need in our task a fine-grained classification, which is by definition (and also due to the specificity of our dialogue classes) harder to obtain. in order to fit with our final application (utterance pre-processing for spoken language understanding in a dialogue), we propose to take advantage both from the robustness of meta-class classification and the interest of the fine-grained one. we are then in a specific top-down classification, limited to two levels (da meta-classes and leaf classes). this corresponds to a "local classifier per level" approach which consists of training a multi-class classifier for each level @xcite . this method is known to have two main drawbacks. first, as it is the case with other top-down class-prediction approaches, errors at the higher level are propagated downwards. second, this method ignores parent-child class relationships. however, it presents in our case an important advantage: information provided at each level will be used directly for utterance understanding in the dialogue system. we propose to keep as first-level classes the metaclasses specified in the iso 24617-2 scheme (see section 4): opening, discourse, inform, question, closing. the second level details the inform metaclass in 8 subclasses: init state, init remediation, bad new state, bad new remediation, current state, current remediation, social interaction, explication. ## reducing the number of classes as explained above, we have tried in our approach to use fine-grained classes, at a precise semantic level. this is of course extremely interesting in the perspective of helping the understanding mechanisms of a dialogue system. but of course, this introduce a new level of difficulty. we have tried to reduce this complexity by defining more general classes, factorizing some of the dialogue acts used previously. in this experiment, we used the following classes: remind that this experiment concerns a classifier for 7 classes. it is therefore not possible to compare directly these results with those of the 13 classes classifiers. the decision of the choice for the final classifier depends then on the classes granularity level we consider optimal, taking into account the performance. the 13 classes flat classifier accuracy is 73.7% (63% balanced accuracy). on its site, the level-2 classification (with the 8 finergrained classes) has an accuracy of 77.2% (71% balanced accuracy). this is to be compared, all things equal, with ## discussion classifying da is in itself a difficult task when trying to take into account a fine level of precision (therefore a high number of classes). most of the experiments reported in the literature focus on general dialog acts, typically the 43 classes for swda that correspond to their general function played by the utterance in the dialogue (question, assessment, backchannel, etc.). when using such general das, automatic classification reaches easily a good accuracy. however, the task becomes difficult when entering into finer-grained das (required when using such classification as a pre-processing step for dialogue systems). in this case, classes correspond to a specific informational content much more difficult to identify than classical functional dialogue acts (in particular because of the natural variability of language). the complexity of this task is reflected in the medium level of inter-annotator agreement we obtained. the first strategy we explored consists in classifying directly our 13 classes. our results are rather good, reaching an accuracy of 73.8%, which is not so far from the average accuracy of the simpler functional classification (75%). xgboost with a feature set involving, on top of n-grams, higher level linguistic, lexical and syntactic features led to the best performance in comparison with 6 other classifiers. these results confirm the importance of the context (in our case the previous da), which improves by 12 points other feature sets. they also show the interest of taking into account linguistic features. lexical features, which are domain-dependent (frequency of medical terms, distinction between pathological terms from others), improve by 2 points the results obtained with contextual features. but note that the med feature set also contains discourse-level information by integrating the count of new referents in the utterance. this feature plays an important role by distinguishing the phases describing patient's state from others: new terms are mainly introduced during the description of the initial state of the patient as well as the description of the bad new. on their side, higher level linguistic features (richer lexical description with the modifiers feature, more complex syntactic constructions with clauses feature) also improves by 2 more points the results with the context and the terminological features. as explained above, these features play a role by distinguishing sequences during which language is more elaborated than others. typically, descriptive or explicative utterances fall into this category. this is an interesting result not only because it improves the performance, but also because we can interpret directly the role of such features in the model. the second strategy we explored in this paper, relying on a hierarchical classification in two steps, shows the interest of mixing generic and specific classes. the results obtained in the first step, classifying 5 generic das, outperform the state-of-the art with an accuracy at 94.2% (obtained by the linear regression model with k-best feature selection) and a balanced accuracy at 89%. in this task, the context feature cannot be used directly, taking into account the class factorization. we can see that linguistic features, as for flat classification, play a primordial role, outperforming in all cases the other feature sets. this is due to the fact that the majority class, inform, gathers all classes possibly impacted by lexical richness and syntactic complexity. let's note moreover that these very good results are very interesting in the perspective of dialogue control: 3 of the 5 classes (opening, discourse, closing) can be directly used in order to generate a response of the dialogue system without entering into a specific processing. the second step of the classification consists in refining the inform class, identifying classes corresponding at the same time to dialogue acts and semantic information (states, remediation, social interaction and explication). in this case again, random forest gives the best classifier. as with all other types of classification, involving linguistic features leads to the best results, which confirms the interest of such feature set whatever the type of dialogue act classification. the final accuracy reaches 77.2% (balanced accuracy 71%) improving the average level of fine-grained da classification. ## conclusion da classification usually relies on low-level features. we have shown in this paper that bringing high level linguistic features in the model improves all types of classification (flat or hierarchical). in particular, we have shown that this approach outperforms the state-of-the-art for generic da classification (corresponding to the level-1 classification in our experiment). even more interestingly, we have shown that this feature set also leads to a very good accuracy for the more complex task of fine-grain da classification. these results open the way to new solutions for language understanding in the context of task-oriented dialogue systems: classifying fine-grained das offers the possibility to precisely identify the semantic frames to be used in the semantic representation of the dialogue.