Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media Data (2022)

Abstract: Automatically detecting the intent of an utterance is important for various downstream natural language processing tasks. This task is also called Dialogue Act Classification (DAC) and was primarily researched on spoken one-to-one conversations. The rise of social media has made this an interesting data source to explore within DAC, although it comes with some difficulties: non-standard form, variety of language types (across and within platforms), and quickly evolving norms. We therefore investigate the robustness of DAC on social media data in this paper. More concretely, we provide a benchmark that includes cross-domain data splits, as well as a variety of improvements on our transformer-based baseline. Our experiments show that lexical normalization is not beneficial in this setup, balancing the labels through resampling is beneficial in some cases, and incorporating context is crucial for this task and leads to the highest performance improvements 7 F1 percentage points in-domain and 20 cross-domain).

## introduction the rise of social media and digital assistants has led to new forms of communication, where an enormous amount of data is available, and automatically understanding this has become an important quest. automatically identifying the intent of an utterance is therefore highly relevant for automatically interacting with humans (i.e. chatbots), or to analyze people's behaviour online. this task is also called dialogue act classification (dac). an example of two social media utterances annotated for dac is shown in ## results all results reported are average macro-f1 scores over 5 random seeds unless mentioned otherwise. as mentioned in section 3, we always used the indomain dev-set for model picking, as well as for hyperparameter tuning. we first evaluate each of our proposed improvements (section 5.1-section 5.3). then, we attempt to combine our methods (section 5.4), and confirm our findings on the test data (section 5.5). we use almost stochastic order (aso) for significance testing @xcite as implemented by @xcite over the random seeds, and with an epsilon (ϵ) smaller than 0.5 we reject the null hypothesis. ## analysis as our standard deviations were relatively small and to simplify our analyses (and for computational efficiency), all results in this section are obtained over one seed. ## conclusion firstly, we found that lexical normalization does not constitute a stochastically dominant feature for cross-domain applications, but rather had a negative effect on f1 performance. by applying lexical normalization, the performance dropped for both our in-domain data, while staying the same for our cross-domain dataset. additionally, while it decreased the standard deviation for our in-domain model, it almost doubled the standard deviations for our cross-domain model. we can therefore state that lexical normalization does not improve the robustness nor increase the f1 score of either in-domain or cross-domain dac model when trained on social media data. secondly, we have seen that multinomial resampling is a stochastically dominant feature in isolation with regard to increasing the f1 score for both in-domain and cross-domain. however, for cross-domain applications it increased the standard deviation drastically, while maintaining the same standard deviation for in-domain usage. used in combination with context for cross-domain applications, we were able to both increase the f1 score marginally, while reducing the standard deviation from 0.92% to 0.42%. we can therefore conclude that multinomial resampling can increase the performance and robustness of a dac model for cross-domain applications when combined with context as a feature, but should not be included for in-domain models. thirdly, we have observed that context has been the most significant contributing factor to the large performance increase of both in-domain and crossdomain models. we have shown that additional context elements in our setup increase robustness and constitutes a stochastically dominant feature compared to fewer context elements. improving the f1 scores by 7.28 percentage-points for in-domain and 21.23 percentage-points for crossdomain, while also reducing the standard deviation to 0.65% for in-domain and 0.92% cross-domain, we have proved that context can improve the robustness of dac models for cross-domain as well as in-domain applications. e. speech situation online, most probably the old data is typed from a keyboard, whereas the reddit part of the data might come from a larger variety of devices. f. text characteristics there could be a variety of noise in the utterances, as well as utterances that contain mainly canonical text. no filtering on style was done.