Evaluating Pretrained Transformer Models for Entity Linking in T ask-Oriented Dialog (2021)

Abstract: The wide applicability of pretrained transformer models (PTMs) for natural language tasks is well demonstrated, but their ability to comprehend short phrases of text is less explored. To this end, we evaluate different PTMs from the lens of unsupervised Entity Linking in task-oriented dialog across 5 characteristics– syntactic, semantic, short-forms, numeric and phonetic. Our results demonstrate that several of the PTMs produce sub-par results when compared to traditional techniques, albeit competitive to other neural baselines. We find that some of their shortcomings can be addressed by using PTMs fine-tuned for text-similarity tasks, which illustrate an improved ability in comprehending semantic and syntactic correspondences, as well as some improvements for short-forms, numeric and phonetic variations in entity mentions. We perform qualitative analysis to understand nuances in their predictions and discuss scope for further improvements.

## introduction in task-oriented dialog systems, entity linking (el) is the process of disambiguating a detected entity mention (aka. slot) in a user utterance to a canonical entry in a knowledge base (kb). el is a crucial step in building robust dialog systems, especially when dealing with domain-specific entities, e.g., a chatbot for food ordering or a voice assistant for medical assistance. popular open-source conversational ai platforms such as deeppavlov @xcite , mindmeld @xcite and rasa @xcite maintain a kb of canonical entries, each consisting of a title, optionally with aliases (i.e., alternate usages) for the task of entity linking. detected entities from user utterances, often with spelling and automatic speech recognition (asr) errors, are then mapped to those canonical entries through text classification or similarity matching techniques. 2 previous works @xcite @xcite have proposed context-aware classification techniques for el, wherein the context surrounding the slots is leveraged to ascertain canonical names. however, such approaches fall short due to (i) their reliance on large training/fine-tuning sets and associated annotation costs (ii) requirement to re-train the classifiers with every change in kb entries. alternatively, a more popular paradigm is to model el as a matching problem by transforming entities into vectors, and using a similarity function such as cosine distance to find the closest canonical entry. el systems typically rely on textual n-gram features modeled by ranking algorithms such as bm25 @xcite implemented as part of search engines such as elasticsearch. 3 to capture semantic similarity within such systems, one needs to tediously engineer feature sets and collect synonyms or aliases for each kb entry, leading to a lot of manual effort and development cost. recently, pretrained word embeddings have had much success in capturing entity correspondences @xcite by addressing aforementioned shortcomingsoff-the-shelf usage without reliance on training data and flexibility to expand kbs without retraining. @xcite presents a detailed account of different deep learning based representations and modeling choices for the el task, showing the advantages of using them over traditional systems. more recently, transformer-based ptms like bert @xcite have excelled for entity linking when entities are in the form of tabular data without much additional context @xcite @xcite . however, their ability to understand nuances in linking short spans of free-form text is not thoroughly tested, especially for domain-specific entities with minimal context. in this work, we investigate and analyze how different ptms behave in such settings, when compared to widely adopted neural and non-neural models ( § 2). to probe model behaviours on examples with different characteristics, we curate and benchmark evaluation datasets of various sizes that each contain a subset of those characteristics ( § 3). lastly, we present qualitative as well as quantitative analysis of the predictions of various models, which shows that while pretrained models fine-tuned for text-similarity tasks perform the best overall, there is room for improvement ( § 4). ## models in this section, we provide a brief detail of the different pretrained transformer models (ptms) as well as the 5 baseline models (3 neural and 2 non-neural) used in our benchmarking process. we categorize ptms under consideration into 4 different types to understand the usefulness of different pre-training strategies, number of parameters and inference times. we adopt the model nomenclature from huggingface foot_3 @xcite and refer the reader to @xcite and @xcite for more comprehensive account on these different types of ptms and their utility. we categorize the ptms as follows: type-i pretrained general-purpose transformer language models which are base-sized. these include bert-base-cased @xcite , roberta-base @xcite and mpnet-base @xcite . type-ii parameters-reduced models which are also trained for language modeling tasks through different parameter reduction techniques. these include albert-base-v2 @xcite , distilbert-base-cased @xcite , distilroberta-base @xcite , and minilm-l6-uncased @xcite . reimers and gurevych (2019) fine-tuned some of the type-i and type-ii models on a variety of datasets annotated for textual similarity tasks foot_4 . we select their all-* models which were fine-tuned with more than 1 billion textual pairs and were designed as general purpose textual similarity models. these include all-distilroberta-v1, all-mpnet-base-v2 and all-minilm-l6-v2. ## datasets we utilize both in-house and publicly available corpora to curate datasets in english for the entity linking task-mindmeld blueprints dataset foot_11 (mm bp) along with word and character level misspelled versions of this data (mm bp-word and mm bp-char), re-purposed open-domain qa datasets like complexwebquestions (complwq) @xcite and mkqa (mkqa) @xcite , acronym identification dataset (acri) @xcite , and an in-house dataset of asr mis-transcriptions for person names (asr-mis). more details on the dataset curation process is provided in appendix a. to probe model behaviours further, we manually annotate 1.3k queries pooled from all of these datasets into our 5 predefined categories as follows (with their sample sizes) -semantic (#294), syntactic (#408), short-forms (#310), numerals (#125) and phonetics (#200). examples from these sets are presented in figure 1 . we use precision@1 (p@1) and precision@5 (p@5) as our benchmarking metrics and conduct all our experiments using the publicly available mindmeld framework foot_12 . unless otherwise stated, we do not include any aliases alongside canonical titles for matching kb entries and utilize all known aliases as our test queries. we disregard any canonical descriptions as they are not always available and procuring them may have significant annotation costs. ## results & analysis ## conclusion given the success of ptms for various nlp applications @xcite , we evaluate the ability of these models to understand short spans of text for unsupervised entity linking in task-oriented dialog systems by curating a large dataset and comparing their results against traditional n-gram systems. we further analyze the performance of these models across 5 different characteristicssyntactic, semantic, abbreviations & short-forms, numeric and phonetic matches. our results demonstrate that these models, when fine-tuned on a semantic similarity task, comprehend syntactic and semantic differences in short phrases better than their other variants. however, their performance is lacking -particularly for abbreviations and queries with speech recognition errors -with the best performing models averaging at 53.4% p@1 and 64.0% p@5 across the different datasets. for future work, with the goal of creating a generic model for the unsupervised el task, we plan to improve these models through task-adaptive fine-tuning techniques with our curated datasets.