Contextual Domain Classification with Temporal Representations (2021)

Abstract: In commercial dialogue systems, the Spoken Language Understanding (SLU) component tends to have numerous domains thus context is needed to help resolve ambiguities. Previous works that incorporate context for SLU have mostly focused on domains where context is limited to a few minutes. However, there are domains that have related context that could span up to hours and days. In this paper, we propose temporal representations that combine wall-clock second difference and turn order offset information to utilize both recent and distant context in a novel large-scale setup. Experiments on the Contextual Domain Classification (CDC) task with various encoder architectures show that temporal representations combining both information outperforms only one of the two. We further demonstrate that our contextual Transformer is able to reduce 13.04% of classification errors compared to a non-contextual baseline. We also conduct empirical analyses to study recent versus distant context and opportunities to lower deployment costs.

## introduction voice assistants such as amazon alexa, apple siri, google assistant and microsoft cortana provide a wide range of functionalities, including listening to music, inquiring about the weather, controlling home appliances and question answering. to understand user requests, the spoken language understanding (slu) component needs to first classify an utterance into a domain, followed by identifying the domain-specific intent and entities @xcite , where each domain is defined for a specific application such as music or weather. in commercial systems, the number of domains tend to be large, resulting in multiple possible domain interpretations for user requests @xcite . for example, "play american pie" can be interpreted as either playing a song or a movie. also, "what does your light color mean?" can be classified as question answering, or as a complaint which does not necessarily require a meaningful response. multiple prior works have attempted to incorporate context in slu to help resolve such ambiguities. however, these works often report results on datasets with limited amount of training data @xcite @xcite , or resort to synthesize contextual datasets @xcite that may not reflect natural human interaction. furthermore, the majority of these works focus on domains where session context is recent and collected within a few minutes. though this setup works well for domains that bias towards immediate preceding context such as communication @xcite and restaurant booking @xcite , there are also domains that have useful context spanning over hours or even up to days. in the smarthome domain, it is natural for users to turn on t.v., watch for a couple of hours and then ask to turn it off. in the notifications domain, users setup alarms or timers which occur hours and days away. we hypothesize that distant context, if properly utilized, can improve performance in instances where recent context cannot. in this paper, we propose temporal representations to effectively leverage both recent and distant context on the contextual domain classification (cdc) task. we introduce a novel setup that contains both recent and distant context by including previous 9 turns of context within a few days, so that context not just come from minutes but can also come from hours or days ago. we then propose temporal representations to indicate the closeness of each previous turn. the key idea of our approach is to combine both wall-clock second difference @xcite and turn order offset @xcite so that a distant previous turn can still be considered as important. we conduct experiments on a large-scale dataset with utterances spoken by users to a commercial voice assistant. results with various encoder architectures show that combining both wall-clock second difference and turn order offset outperforms using only one of them. our best result is achieved with transformer of 13.04% error reduction, which is a 0.35% improvement over using only wall-clock second difference and 2.26% over using only turn order offset. to understand the role of context in cdc, we conduct multiple empirical analyses that reveal the improvements from context and discuss trade-offs between efficiency and accuracy. to summarize, this paper makes the following contributions: ## methodology in this section, we describe our model architecture in section 3.1 and our proposed temporal representations in section 3.2. ## results in this section, we first describe our experimental setup in section 4.1, present our main results in section 4.2, followed by our analyses in section 4.3. ## conclusions we presented a novel large-scale industrial cdc setup and show that distant context also improves slu. our proposed temporal representations combining both wall-clock and turn order information achieved best results for various encoder architectures in a hierarchical model and outperforms using only one of the two. our empirical analyses revealed how previous turn helps disambiguation and showed opportunities on reducing deployment costs. for future work, we plan to explore more turn features such as responses, speaker and device information. we also plan to apply temporal representations on other tasks, such as intent classification, slot labeling, and dialogue response generation.