A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO -Standard Dialogue Act Tagging (2022)

Abstract: Dialogue Act tagging with the ISO 24617-2 standard is a difficult task that involves multi-label text classification across a diverse set of labels covering semantic, syntactic and pragmatic aspects of dialogue. The lack of an adequately sized training set annotated with this standard is a major problem when using the standard in practice. In this work we propose a neural architecture to increase classification accuracy, especially on low-frequency fine-grained tags. Our model takes advantage of the hierarchical structure of the ISO taxonomy and utilises syntactic information in the form of Part-Of-Speech and dependency tags, in addition to contextual information from previous turns. We train our architecture on an aggregated corpus of conversations from different domains, which provides a variety of dialogue interactions and linguistic registers. Our approach achieves state-of-the-art tagging results on the DialogBank benchmark data set, providing empirical evidence that this architecture can successfully generalise to different domains.

## introduction language understanding is a fundamental component of any conversational system, as it impacts its abilities to correctly recognise a user's communicative functions and act accordingly. dialogue act (da) tagging is a crucial step of this understanding process, particularly in an open-ended conversational setting, as it informs the system on the users' beliefs, desires, intentions and actions. task:agreement early dialogue applications usually adopted a list of mutually exclusive and task-specific da tags which represented the different functions that the system performed, acting essentially as intent labels. these taxonomies were also one-dimensional, featuring mutually-exclusive tags which did not account for the complexity of the dialogues. the following example from @xcite clarifies the importance of multi-dimensionality in da tagging: s: can you tell me what time is the first train to the airport on sunday morning? a: on sunday morning the first train to the airport is at 5.32. s: thank you! according to @xcite , the third utterance has two separate communicative functions, as the speaker s is expressing gratitude towards the addressee a (social dimension), while at the same time informing them on their understanding of the train schedule (feedback dimension). in addition to this, da taxonomies have also typically lacked a hierarchical organisation of tags, which makes it difficult for a classifier to capture the high-level mutual relationships between dialogue tags @xcite . in an attempt to address these problems, an official iso standard taxonomy, iso 246170-2, was introduced in @xcite : this taxonomy is domain-independent, hierarchical and multidimensional, and well-suited for open-ended natural language understanding. however, much work in dialogue systems still uses other taxonomies, in part we believe because of the lack of an adequately-sized data set annotated with the standard, which makes it difficult to train a classifier for the taxonomy. some authors have proposed automated mappings of old resources to the new iso standard; however, these works are still limited in scope, focusing either on heavily imbalanced data sets @xcite or subsets of the iso 24617-2 taxonomy that are insufficient for real-life conversational scenarios @xcite . da tagging with the iso taxonomy is also an intrinsically difficult task, as it requires handling multiple different dimensions and a collection of different fine-grained tags which differ in semantic, syntactic and contextual aspects. the open-ended nature of the taxonomy provides an additional challenge, as most existing da-annotated resources tend to have a bias towards specific topics or discussion styles, which hinders the model's capabilities to generalise to unseen conversations. in this work, we introduce a neural architecture optimised for da tagging with a subset of the iso standard taxonomy. our model combines syntactic, semantic and contextual information and leverages the hierarchical dependencies across labels to improve the classification accuracy, especially on low-frequency fine-grained tags. we combine existing da-annotated data sets and map them to a subset of the iso 24617-2 taxonomy to obtain an adequately-sized training set, taking advantage of existing mappings in the literature @xcite and novel conversational resources such as mastodon @xcite and dailydialog @xcite , whose taxonomies easily map to the iso standard. we also experiment with the addition of online discussions and debates data from the internet argument corpus v2.0 @xcite , in order to increase the system's understanding of opinionated and con-textual tags. experimental results show that our approach achieves state-of-the-art classification accuracy on the dialogbank @xcite test set. we also provide additional experiments that delve into the details of the training process, including ablation studies and an analysis of the extent to which the different corpora that we utilised contribute to the network's performance. finally, we share our code and our mapped data set foot_0 , in order to share these resources with the research community and hopefully encourage further research on dialogue act classification. ## related work the concept of dialogue act (da) has its roots in the seminal works by @xcite and @xcite , who established the theoretical foundations of the speech act theory. a speech act captures an utterance at the level of its illocutionary force. many subsequent works started referring to speech acts in a conversational setting as dialogue acts, and investigated possible taxonomies of dialogue acts. these early taxonomies were flat (there was no distinction between coarse-grained and finegrained tags), mono-dimensional (each and every utterance had only one dialogue act tag) and usually task-specific rather than domain-independent. examples of these early taxonomies include the damsl taxonomy @xcite , which was used for the annotation of the switchboard and mrda conversational corpora, the hcrc coding manual @xcite , which was used to annotate the maptask corpus, and the verbmobil annotation scheme @xcite , which was used for the annotation of the homonymous corpus. an interest in formalising these taxonomies into a more rigid theoretical framework arose in the early 2000s, with @xcite analysing existing da taxonomies and investigating a rigorous definition of dialogue acts. @xcite provided one of the first formal definitions of dialogue act as "a unit in the semantic description of communicative behaviour, produced by a sender and directed at an addressee, specifying how the behaviour is intended to influence the context through understanding of the behaviour". the authors combined existing taxonomies such as damsl and dit @xcite into a new taxonomy called dit++ @xcite , which aimed at being a truly open-ended, domain-independent and theoretically sound tax-onomy. the fifth version of the dit++ taxonomy became the official iso standard for dialogue act classification @xcite . a potential advantage of using the standard is that multiple corpora can be used to construct a larger training set, or cross-domain training sets suitable to cover a wide rage of dialogue tasks; nonetheless, the iso 24617-2 taxonomy has yet to be fully adopted, with many works still using damsl as their target taxonomy @xcite , or introducing entirely novel taxonomies customtailored for specific tasks @xcite . while this is partly due to the complexity of the standard, we believe a significant obstacle is the lack of adequately-sized data sets to train a classifier. some efforts have been made to convert existing resources to the new taxonomy @xcite proposed an automated, albeit partial, mapping of the damsl taxonomy to the iso standard, @xcite extended their work providing partial mappings for the ami, maptask, oasis and verbmobil taxonomies, and @xcite provides a mapping from the lego annotation scheme to the iso one. there are some planned corpora entirely annotated with the iso standard, such as adele @xcite or dbox @xcite , which may become valuable tools to work with the taxonomy. however, these resources are still not publicly available at the time of writing. some interesting corpora were released in recent years which, while not being entirely iso compliant, adopted da taxonomies which can be easily mapped to the standard. these include dailydialog @xcite , mastodon @xcite and midas @xcite among others. automatic da tagging was initially formalised as a text classification task by @xcite , who presented a hidden markov model for the classification of the switchboard data set. since then, many different approaches have been proposed to tackle the task, including rule-based systems @xcite , conditional random fields @xcite and support vector machines @xcite . more recent works shifted their focus to artificial neural networks, which have been proved to be very effective for text classification tasks; many of these models are built around bi-lstm/gru cells with crf as a top layer, due to this architecture's ability to capture long-term contextual dependen-cies in dialogue @xcite . many architectures also rely on transformers, usually combined with pre-trained sentence embeddings @xcite or some form of attention mechanism @xcite . the vast majority of these models are designed for flat hierarchies of tags, with little to no emphasis on the multi-dimensional nature of dialogue acts. @xcite annotated a subset of the tradr corpus of robot-assisted disaster response team communications with three dimensions of the iso standard (general (task), social and turn management) and trained various neural classifiers for the task, including cnn, lstm and ffn; while their work does take advantage of the multi-dimensional aspect of the iso taxonomy, the scope of their research is limited by the size of the resource and the emphasis on the disaster response domain. a number of recent works started taking advantage of the hierarchical structure of iso communicative functions: @xcite introduced a hierarchical neural model for one-dimensional da tagging (they only consider the social and task dimensions of the standard and combine them into a single core dimension), while @xcite proposes a hierarchical and multi-dimensional approach for the spanish corpus dihana. @xcite apply a number of statistical machine learning algorithms, such as xgboost and random forests, to annotate french medical data with a subset of the iso standard; their approach separates the classification into two hierarchical steps to increase the accuracy of the model. @xcite proposed a multi-dimensional and domain-independent approach to da tagging, and also investigated hierarchical da tagging through a tree-like structure of svm classifiers; however, the model was limited in scope and accuracy and did not take into account contextual tags such as answer, agree/disagree, etc. @xcite utilised an end-to-end hierarchical network with cascading outputs and maximum a posteriori path estimation to classify all the layers of the general (task) semantic dimension; while their architecture handles the whole taxonomy of task communicative functions, it lacks support for additional dimensions of the standard such as social, turn management, etc. their model also fails to capture the domain-independent nature of the taxonomy, as its performance degrades with the addition of out-of-domain data such as the conversations from the lego-iso corpus. multi-label hierarchical text classification has been successfully addressed for other text classification tasks in the literature, such as fine-grained sentiment analysis @xcite or topic classification @xcite . the latter work proposed two different models to solve hierarchical, multi-dimensional topic classification on news articles; we adapt a similar approach to include prior information on the hierarchical correlation among labels in our model. ## results and discussion in this section we present the results of our experimental study. we evaluate the performance of our model on two test sets, namely the test split of gdc and the dialogbank corpus. we provide average micro-f1 and macro-f1 scores for our model, with the former providing a measure of its raw accuracy and the latter providing a better metric for how well low-frequency tags are correctly classified. the dashnet model was able to correctly annotate a large portion of the test split of the gdc data set. moreover, it also shows promising results on the dialogbank test set, highlighting good generalisation when classifying out-of-domain data. ## conclusion we have presented a multi-dimensional, crossdomain neural architecture for iso-standard dialogue act tagging, which leverages the hierarchical nature of the standard as well as grammatical, lexical and contextual information of the input utterances. we trained the model on a general dialogue corpus composed of different resources mapped to the iso 24617-2 taxonomy, and showed how our model achieves state-of-the-art performance on out-of-domain data, which highlights its generalisation capabilities. the code and gdc data set have been released so as to help advance research on this topic. in the future, we plan to expand our work by covering more iso-annotated corpora once they become available, as well as extending the context encoder module of our architecture to cover a wider context of dialogue and da history. we also plan to experiment with different embedding mechanisms, such as bert @xcite and roberta @xcite .