The impact of domain-specific representations on BERT -based multi-domain spoken language understanding (2021)

Abstract: This paper provides the first experimental study on the impact of using domain-specific representations on a BERT-based multi-task spoken language understanding (SLU) model for multi-domain applications. Our results on a real-world dataset covering three languages indicate that by using domain-specific representations learned adversarially, model performance can be improved across all of the three SLU subtasks domain classification, intent classification and slot filling. Gains are particularly large for domains with limited training data.

## introduction spoken language understanding (slu) is a key task in voice-controlled devices, such as amazon alexa or google home. it is often divided into the two subtasks intent classification (ic) determining the user intent and slot filling (sf) extracting semantic constituents. for instance, given an utterance "play madonna", ic should determine play-music as the intent, while sf should label "play" and "madonna" as other and artist slots, respectively. the third subtask, domain classification (dc) which classifies user utterances into different domains, is sometimes required, especially in large-scale industry applications. dc helps when the data comes from different domains and the slu labeling results require some domain-specific postprocessing steps. aiming at reducing model building efforts, in this paper, we focus on single multi-task models solving the three subtasks of dc, ic and sf jointly for multi-domain data. traditionally, these models learn shared-representations for all domains (e.g. @xcite ; @xcite ; @xcite ). however, using only domain-shared representations may limit model performance as it ignores potentially useful domain-specific knowledge. for instance, let us consider two user requests "play madonna" and "play star wars", belonging to the music and video domains, respectively. in this case, the carrier phrase "play" is shared while the slot values "madonna" and "star wars" are specific to music and video, respectively. thus, both domain-shared and domain-specific representations could potentially be useful. in order to shed a light on whether (missing) domain-specific knowledge may play an impact on the model performance, we carry out a wide range of experiments on real-world data comparing the performances of dc-ic-sf multi-task models with and without domain-specific representations. to assure that the models used in the experiments are close to the current state-of-the-art systems, we adapt one of the most recent bert-based languageadversarial approaches for ic-sf @xcite to our problem. in particular, we use its adversarial architecture to learn domain-shared and domainspecific representations, and extend the model to solve dc in addition to ic and sf. our contribution in this paper is studying the impact of using domain-specific representations on a modern multi-task dc-ic-sf model for multidomain data. our experiments on a real-world dataset covering three languages (german, english, japanese) indicate that the domain-specific representations can improve model performance across all of the three slu subtasks, especially for domains with limited training data. ## related work a majority of existing work in slu has focused on modeling ic and sf only, for which mostly dnnbased joint models are currently used (e.g. @xcite ; @xcite ; chen et al. multi-domain joint slu models have been explored aiming to solve all three tasks via multi-task learning (e.g. @xcite , @xcite , @xcite ). however, none of these studied the impact of domain-shared and domain-specific representations on model performance. while some previous work followed the idea of leveraging domain-specific representations, we are not aware of a previous study using a full bertbased multi-domain slu model with domainadversarial learning. @xcite explored adversarial training in multi-domain slu, but focus on sf only. in another work, @xcite applied adversarial training to obtain localespecific and locale-agnostic features for the dc task. more recently, @xcite explored domain-specific parametrization for multi-domain slu in a cascade approach, i.e. a domain classifier is applied as a first step instead of a completely joint system like our focus. ## method in the following, we first describe the multi-domain baseline slu model, which uses bert as a single shared feature extractor, and subsequently the adversarial approach, where additionally a domainspecific feature extractor is used for each domain. ## results in our experiments video is considered as a new domain, where initially few domain data samples are available, and data amounts are growing over time. in particular, we consider samples with sizes of 100, 500, 1,000 and 5,000 utterances for the new domain. for each experiment, the total data amounts for the domains music and books are kept constant at 10,000 and 5,000 utterances, respectively. for each considered video sample size, we combine the corresponding video training and development data with the other domain's training and development data, respectively. we then train i) a multi-domain slu model as described in section 3.1, and ii) an adversarial slu model as described in section 3.2, which are subsequently applied on the test datasets for all domains. the results are presented in overall, the domain-adversarial approach con-sistently outperforms the baseline model across all tasks and languages, except for some small fluctuations for the dc task in the music domain, which has the largest data amounts in our scenario. in turn, dc performance is improved slightly for books and greatly for video. in particular, averaged across languages and sample sizes, dc accuracy is improved by 1.21% and 89.37% relative for the books and video domains, respectively. performance gains in the video domain are particularly large for smaller samples sizes. for the ic and sf tasks, the largest gains are also achieved for the video domain. averaged across languages and sample sizes, we achieve relative improvements in ic and sf of 29.0% and 27.16%, for the video domain. again, the largest relative improvements are achieved when only a few video samples are available. the results suggest that the adversarial approach is particularly useful to boost performance of domains with a limited amount of data, which could be new or low-frequency domains. while the highest gains are achieved when only few domain data is available, consistent gains in ic and sf performance are achieved for all domains and data sizes, with gains being larger for the sf task. a potential reason could be that the amount of relevant domain-specific information is higher for sf than for ic, and thus larger gains are possible by adversarial training. note that domain-shared information can also be leveraged by the baseline model (though it may not be domain-agnostic). the results also reveal that there are language-specific differences. specifically, the domain-adversarial training approach seems to be particularly useful for japanese. averaged across sample sizes, relative improvements in slot filling performance of 15.75% and 10.6% for music and books, respectively, are achieved. ## conclusion we studied the impact of using both domainspecific and domain-shared representations vs. using only domain-shared representations on the application of multi-domain slu. in particular, we compared a baseline which uses bert as a single shared feature extractor to learn domain-shared representations to an adversarial model which additionally uses a domain-specific feature extractor for each domain to learn domain-specific representations in addition to the standard domain-shared representations. our results on a real-world dataset covering three languages indicate that by using a domain-adversarial approach, model performance can be improved across all of the three slu subtasks. performance gains were particularly large for the use case of bootstrapping a new domain, where little target domain data are available.