Optimizing Entity Resolution in Voice Interfaces: An ASR -Aware Entity Reference Expansion Approach (2024)

Abstract: This paper tackles the challenges presented by Automatic Speech Recognition (ASR) errors in voice-based dialog systems, specifically, their adverse impact on Entity Resolution (ER) as a downstream task. Navigating the equilibrium between accuracy and online retrieval’s speed requirement proves challenging, particularly when limited data links the failed mentions to resolved entities. In this paper, we propose a entity reference expansion system, injecting pairs of failed mentions and resolved entity names into the knowledge graph, enhancing its awareness of unresolved mentions. To address data scarcity, we introduce a synthetic data generation approach aligned with noise patterns. This, combined with an ASR-Error-Aware Loss function, facilitates the training of a RoBERTa model, which filters failed mentions and extracts entity pairs for knowledge graph expansion. These designs confront obstacles related to ASR noise, data limitations, and online entity retrieval.

## introduction in the domain of voice-based dialog systems, the inherent inaccuracies within automatic speech recognition (asr) pose significant impediments to downstream tasks. specifically, as the transcribed input undergoes processing by a natural language understanding (nlu) component to extract structured data such as entity mentions, errors in asr frequently propagate to the subsequent component in a dialog system -entity resolution (er). er is the process of linking labeled mentions to a knowledge base, and the reliance on asr accuracy exacerbates the intricacy of this task. further complicating matters is the imperative for stringent resource optimization, mandated by the latency requirements associated with deploying er systems on devices. within this context, the most pragmatic workaround, namely token-based matching, may encounter limitations when confronted with noisy or ambiguous entity mentions. for example, a token-based system might proficiently recognize "flying gorilla" but could falter when dealing with semantically or phonetically akin phrases such as "frying gorilla" or "flying gloria." these challenges often emanate from asr errors, underscoring the necessity for nuanced solutions in the development of formal voice-based dialog systems. to address these challenges, this paper introduces an enhanced entity reference enrichment system for the knowledge graph. our offline model, depicted in figure 1 , utilizes a synthetic data generation pipeline that augments all entity names to replicate error patterns observed in live traffic. this approach addresses data scarcity issues and enables fine-tuning of a roberta-based encoder using a cross entropy loss that is sensitive to asr-induced inaccuracies, capturing both semantic and phonetic subtleties. the model specifically encodes and generates pairs between previously failed mentions and successfully resolved entity names, which are injected back into the knowledge graph to improve its handling of historically unresolved mentions while maintaining low latency in our industrial retrieval pipeline. although this approach focuses on errors previously encountered, it captures a significant proportion of common error patterns. while a more dynamic system capable of addressing new asr errors could involve runtime vector searches, the required infrastructure changes and potential latency impacts make our current method a practical short-term solution, setting the stage for future enhancements. the contributions of this paper are three-fold: firstly, it introduces a synthetic data generation approach to train the model against upstream patterns of noises and reduce manual labeling. secondly, our asr-error-aware loss function enhances the roberta model's performance in handling asrinduced inaccuracies. finally, the paper proposes a knowledge graph (kg) injection strategy that can be integrated into the runtime pipeline without modifying the existing online retrieval strategy. these contributions collectively address challenges related to training data, asr noise, and online processing. ## related work early initiatives for entity resolution tasks, such as neural entity linking (nel) and entity disambiguation (ed), utilized fully-connected neural networks or long short-term memory (lstm) networks to encode mentions and entity names @xcite . the emergence of deep pre-trained models like bert @xcite and their fine-tuned derivatives, marked a paradigm shift in methodologies for entity linking and entity resolution @xcite . these models typically embed entity mentions and names into a dense vector space, employing architectures such as two-tower designs @xcite , and calculating the semantic similarity between mentions and entities in the knowledge base @xcite . vector search or nearest neighbor search techniques are commonly used for retrieving the best candidate entities. however, they cannot scale in high-latency settings @xcite . innovations like the siamese structure with improved alignment networks proposed by @xcite aim to reduce exhaustive computa-tions. in contrast, we introduce an offline process for entity pair extraction to minimize online latency demands. while wang et al. @xcite ) focused on improving entity retrieval by correcting asr-induced query errors, we utilize a fine-tuned encoder model, notably roberta, to enhance entity retrieval accuracy by expanding the candidate pool, addressing a different facet of the er challenge. the dilemma of scarce labeled data in industrial nlp applications is well-acknowledged, with the lack of manual annotation posing significant constraints. while model transfer and data augmentation are common remedies, our approach leans towards data augmentation. this strategy aligns with our objectives, providing cost-effective control over training data distribution and enabling us to fine-tune our model in a manner that is more reflective of real-world voice-based interactions. ## conclusion in conclusion, our entity reference expansion pipeline, utilizing a fine-tuned roberta model, seeks to enhance entity resolution (er) in voicebased conversational systems. the synthetic data generation approach, which emulates noise patterns, facilitates model training without requiring manual labeling, while the implementation of an asr-error-aware loss function addresses challenges arising from asr-induced noise. furthermore, our knowledge-graph-injection approach, executed offline, ensures the system's robustness while seamlessly aligning with the industry's online retrieval design for swift performance. our developments offer new perspectives in enhancing er solutions, contributing to the ongoing improvement of voice-based dialog systems.