Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding (2023)

Abstract: Leveraging representations from pre-trained transformer-based encoders achieves state-of-the-art performance on numerous NLP tasks. Larger encoders can improve accuracy for spoken language understanding (SLU) but are challenging to use given the inference latency constraints of online systems (especially on CPU machines).We evaluate using a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU compared to using smaller 17M parameter BERT encoders with language-, domain- and task-decoupled finetuning.Running inference with a larger shared encoder on GPU is latency neutral and reduces infrastructure cost compared to running inference for decoupled smaller encoders on CPU machines. The larger shared encoder reduces semantic error rates by 4.62% for test sets representing user requests to voice-controlled devices and 5.79% on the tail of the test sets on average across four languages.

## introduction spoken language understanding (slu) plays an essential role in voice-controlled devices such as amazon alexa, apple siri and google assistant. two commonly studied slu subtasks are intent classification (ic) and slot filling (sf). while ic classifies an utterance into a set of pre-defined intents, sf aims to extract relevant slot information. for example, given an utterance "play madonna" ic should determine playmusic as the intent and sf should detect "madonna" as artist. the two subtasks are often modeled jointly @xcite @xcite . in large-scale slu systems intents can be split into separate domains allowing the same intents to exist in different domains and for domain-specific teams to work independently of each other. consequently, multidomain slu models have been developed that ad- * corresponding author: hueserjh@amazon.de ditionally address the task of domain classification (dc). for example, relating back to the previous example dc should detect music as the domain. in this paper, we focus on a multi-domain system with domain-specific ic+sf models due to its advantages for large-scale slu, such as the support for independent development across domains and the option of updating and deploying only certain domain models instead of the whole system. neural network models for ic+sf and dc tasks typically leverage language representations from finetuned language modeling encoders as features. bidirectional encoder representations from transformers (bert) pre-trained on large corpora achieve state-of-the-art performance in slu @xcite @xcite . larger encoders can give better performance on down-stream tasks after being finetuned @xcite . however, increasing the number of encoder parameters for large-scale online slu systems is challenging due to increased inference latency and infrastructure cost constraints. in this paper we present a method for bringing the accuracy benefits of larger encoders to the users of large-scale slu systems. we propose to share representations from a large frozen multilingual encoder as features for all of the dc and domain-decoupled ic+sf task heads across multiple languages. we compare a 170m parameter (not counting embeddings) shared encoder for four languages (german, italian, spanish, french) to a set of language-, domain-and task-decoupled 17m parameter encoders. for reference, the common bert-base and bert-large models are 87m and 306m parameters not counting the embeddings (110m and 340m parameters with embeddings). as a reference for infrastructure cost we consider inference of the language-, domain-and task-decoupled 17m parameter encoders on cpu machines as gpu inference does not meet the infrastructure cost constraints of the specific industry slu system that our baseline is based on. if the 170m parameter shared encoder inference is run on gpu and representations are cached then the inference latency is neutral compared to running the 17m parameter encoder inference on cpu. the infrastructure cost savings from not having to run a 17m parameter encoder inference for every domain and language on cpu pay for the infrastructure cost of the required gpu instances. in fact, the shared encoder architecture obtains infrastructure cost savings in practice. to summarize, this paper contains the following contributions. we present a novel shared encoder slu architecture that enables the use of a larger encoder to improve accuracy while staying inference latency and infrastructure cost neutral. we run large-scale experiments with both internal data of an industry slu system (23 domains in four languages) and the public massive dataset (18 domains in four languages) @xcite . a 170m parameter shared encoder compared to 17m parameter decoupled encoders reduces semantic error rates by 4.62% for test sets representing user requests to voice-controlled devices and 5.79% on the tail of the test sets on average across four languages. we introduce a light-weight encoder that is trained together with the language-and domain-decoupled task heads to learn representations for training data unseen by the frozen shared encoder. we conduct empirical analyses that study the shared encoder setup in the context of feature expansion and distribution drift across a real-world slu system release cycle. ## related work sharing language encoder representations across tasks via multi-task learning has been a prominent research thread in recent years that has resulted in a vast literature. for example, @xcite evaluate a transformer encoder @xcite as a universal sentence encoder trained across multiple natural language processing tasks without prior self-supervised pre-training. @xcite show that multi-task learning can be leveraged on top of self-supervised pre-training for bert encoders to improve performance across multiple natural language understanding tasks. the multi-domain, multi-task approach to joint domain classification, intent classification and slot filling that we employ in our shared encoder pre-finetuning was already explored by @xcite . using a "text-to-text" approach large language models (llms) have been demonstrated to be multi-task learners with cross-task generalization @xcite @xcite and frozen llms can also be adapted to different tasks via prompt tuning approaches @xcite . multilingual encoder representations for spoken language understanding are often motivated by cross-lingual transfer @xcite . zhang et al. (2021) evaluate a multi-head decoding architecture with a multilingual encoder and language-specific task heads for intent classification and slot labeling that is similar to our proposed shared encoder architecture. the impact of scaling the parameter count in large language models and pre-trained models in general has been explored extensively by @xcite ; @xcite ; @xcite . distillation can be used to close some of the performance gap between smaller and larger language encoders while meeting inference latency constraints @xcite @xcite . but for example @xcite show that using larger distilled encoders reduces the semantic error rates for spoken language understanding which motivates our exploration of a shared encoder architecture to enable encoder scaling. ## methodology our goal is to introduce a larger bert encoder to increase accuracy for an slu system with language, domain and task-decoupled ic+sf and dc models without regressions in inference latency and infrastructure cost. section 3.1 provides context on practical challenges that need to be addressed by a real-world slu system and that we consider for our proposed model architecture. we then describe how encoder representations are shared across languages, domains and tasks and explain our shared encoder model architecture in section 3.2. in section 3.3 we describe the impact on training, inference latency and infrastructure cost. ## conclusions we present a novel shared encoder architecture that enables the use of larger encoders in a realworld slu system while staying inference latency and infrastructure cost neutral. by sharing representations from a larger encoder across languages, domains and tasks the semantic error rates of the slu system can be reduced consistently across languages for test sets representing both the full user request distribution and its tail. our empirical analyses reveal that a light-weight encoder can be used in combination with the shared encoder architecture to avoid retraining the frozen shared encoder for every new feature release.