Entity Contrastive Learning in a Large-Scale Virtual Assistant System (2023)

Abstract: Conversational agents are typically made up of domain (DC) and intent classifiers (IC) that identify the general subject an utterance belongs to and the specific action a user wishes to achieve. In addition, named entity recognition (NER) performs per token labeling to identify specific entities of interest in a spoken utterance. We investigate improving joint IC and NER models using entity contrastive learning that attempts to cluster similar entities together in a learned representation space. We compare a full virtual assistant system trained using entity contrastive learning to a production baseline system that does not use contrastive learning. We present both offline results, using retrospective test sets, as well as live online results from an A/B test that compared the two systems. In both the offline and online settings, entity contrastive training improved overall performance against production baselines. Furthermore, we provide a detailed analysis of learned entity embeddings, including both qualitative analysis via dimensionality-reduced visualizations and quantitative analysis by computing alignment and uniformity metrics. We show that entity contrastive learning improves alignment metrics and produces well-formed embedding clusters in representation space.

## introduction named entity recognition (ner) is a well-studied and fundamental task within natural language understanding (nlu). the performance of a virtual assistant is heavily dependent upon how well ner tasks are handled. mistaken slot predictions result in propagating incorrect information to downstream modules, causing sub-optimal interactions with users of the system. contrastive learning can be used to improve ner model training. contrastive learning attempts to cluster similar inputs closer together in their representation space and repel dissimilar inputs apart. token contrastive learning attracts and repels representations at the token level and was introduced in @xcite for improving performance in few-shot ner tasks. in this work, we apply contrastive learning to improve the performance of a ubiquitous virtual assistant system. we first train a common encoder using contrastive sentence embedding @xcite . next, we incorporate entity contrastive learning, based on @xcite , to better cluster similar entities together in representation space. we train and evaluate joint ic and ner models in 11 domains. for each domain, we evaluate performance with and without an additional entity contrastive loss. we further provide results of an online a/b test that measures user satisfaction and show improved performance when using entity contrastive training. furthermore, we perform a detailed embeddings analysis to determine the effect that the entity contrastive loss function has on entity representations. in particular, we compute alignment and uniformity metrics (wang and isola, 2020) of learned entity representations. finally, we also present qualitative results in the form of t-sne visualizations comparing models with entity contrastive training vs. without. we show that entity contrastive learning improves alignment metrics as well as clustering behavior in representation space. ## virtual assistant system overview fig. 1 shows a schematic overview of a jointly trained ic and ner model that makes up part of the nlu component of a full virtual assistant system. joint ic-ner models are trained separately for each domain. the ic-ner model encodes a sequence of (sub-word) utterance tokens, x 1 , x 2 , . . . , x n , through a transformer encoder architecture, in addition to sub-words that are fed to the encoder, each input token is also flagged as either being recognized or unrecognized via lookup in a large gazetteer, ϕ(•) ∈ {0, 1}, which further undergoes a separate gazetteer-based embedding, ## experimental results we provide experimental results in the following three settings: offline (per domain): we compare 11 domain models trained using entity contrastive learning vs. baseline models without entity contrastive training. all domains that utilize gazetteers are included. offline (full system): we compare a full virtual assistant system trained using entity contrastive learning against a baseline system on a collection of static test-sets. online: we conduct an a/b test using live traffic to compare a full virtual assistant system trained using entity contrastive learning vs. a baseline model that does not. full descriptions of each error metric used for (offline) evaluation are given in appendix a. we provide brief summaries here: semer: semantic error rate reflects the proportion of incorrectly labeled entities and intents. icer: intent classification error rate measures the proportion of misclassified intents irer: intent recognition error rate measures how often predictions contain any mistakes in either entities or intent. ## embeddings analysis we further provide qualitative and quantitative analysis of the entity representations learned by a baseline and contrastive model. the baseline model differs to the contrastive model only by removing the ℓ en t component of the loss function in eqn. (3). ## related work contrastive learning has been applied with tremendous success over the last few years in tasks that process data such as audio @xcite , vision @xcite ) and natural language @xcite . contrastive losses, such as infonce @xcite , build on the original idea of noise contrastive estimation @xcite that learns a data distribution by comparing it against a chosen noise distribution. contrastive representation learning can either be unsupervised @xcite or supervised @xcite . unsupervised or self-supervised approaches have relied upon techniques such as data augmentation @xcite and future self prediction @xcite as a way of ignoring superfluous information to learn better class representations. supervised approaches @xcite incorporate class label information during learning and were introduced to avoid problems with in-batch false positives. in natural language tasks, contrastive learning approaches based on data augmentation techniques have not fared as well compared to their vision counterparts. sim-cse @xcite introduced both unsuper-vised and supervised approaches for learning contrastive sentence embeddings. the unsupervised approach relies solely on varying dropout masks to achieve different representations of the same input sentence, whereas the supervised task uses examples from natural language inference datasets @xcite . rather than learning sentence embeddings, (das et al., 2022) introduced token contrastive learning in the context of improving few-shot learning. our work does not focus on few-shot learning, but instead seeks to evaluate joint ic-ner models trained with entity contrastive learning for the purpose of improving a large-scale virtual assistant system. ## conclusion we presented jointly trained ic and ner models augmented with entity contrastive learning via an additional loss function that attempts to pull similar entities together in representation space, and repel dissimilar entities apart. we provided a comprehensive evaluation of entity contrastive learning within a full virtual assistant system by comparing to baselines in both offline and online (a/b test) experiments. results show that employing entity contrastive learning improves overall error and alignment metrics and produces well-formed embedding clusters in representation space. luke vilnis and andrew mccallum. 2015. word representations via gaussian embedding. in 3rd international conference on learning representations, iclr 2015, san diego, ca, usa, may 7-9, 2015, conference track proceedings. tongzhou wang and phillip isola. 2020. understanding contrastive representation learning through alignment and uniformity on the hypersphere. in international conference on machine learning, pages 9929-9939. pmlr.