Real World Conversational Entity Linking Requires More Than Zero-Shots (2024)

Abstract: Entity linking (EL) in conversations faces notable challenges in practical applications, primarily due to scarcity of entity-annotated conversational datasets and sparse knowledge bases (KB) containing domain-specific, long-tail entities. We designed targeted evaluation scenarios to measure the efficacy of EL models under resource constraints. Our evaluation employs two KBs: Fandom, exemplifying real-world EL complexities, and the widely used Wikipedia. First, we assess EL models’ ability to generalize to a new unfamiliar KB using Fandom and a novel zero-shot conversational entity linking dataset that we curated based on Reddit discussions on Fandom entities. We then evaluate the adaptability of EL models to conversational settings without prior training. Our results indicate that current zero-shot EL models falter when introduced to new, domain-specific KBs without prior training, significantly dropping in performance.Our findings reveal that previous evaluation approaches fall short of capturing real-world complexities for zero-shot EL, highlighting the necessity for new approaches to design and assess conversational EL models to adapt to limited resources. The evaluation frame-work and dataset proposed are tailored to facilitate this research.

## introduction entity linking (el) is the process of detecting and resolving ambiguous mentions of entities in a given text by accurately associating them with their corresponding entries in a knowledge base @xcite . this is a pivotal step in many downstream tasks such as semantic search @xcite @xcite , question answering @xcite , and conversational search @xcite . the significance of el particularly comes to the fore in the realm of conversational systems as it helps to enhance the accuracy and relevance of the information provided to users during a dialogue session. as these systems are becoming increasingly prevalent in various applications, their ability to ground discussions in real-world knowledge is indispensable for maintaining the integrity and usefulness of the system @xcite @xcite . conversations possess characteristics that render common el models suboptimal; e.g., noisy text, informal language, and entity-related information spreading through turns @xcite . however, conversational el has been less explored in prior research and is predominantly focused on techniques and benchmarks for long documents @xcite or stand-alone queries @xcite . el approaches also often assume the existence of ample training data @xcite @xcite , a similar distribution of entities in kb during training and at inference time, and a structurally/textually rich kb for training. these assumptions, however, do not usually hold in real-world el scenarios, especially in a conversational context, making el in practice more challenging. train and deployment of el systems in general poses several other challenges as well. creating an entity-annotated training dataset can be prohibitively exhaustive, or the data might be unavailable due to privacy concerns @xcite . in addition, the distribution of train and test entities might differ as knowledge bases may expand with time, and new entities can be added to the kb which results in an incomplete kb at training time @xcite . lastly, real-world kbs do not often come with dense structural/textual entity information. as a result, zero-shot entity linking @xcite was introduced to address some of these challenges. this setup is aimed to allow disambiguating mentions of previously unseen entities by relying on pre-trained models. in this study, we design an evaluation framework and a dataset, addressing the gap between real-world conversational el and the existing zeroshot el studies, showing that current zero-shot models do not adequately address practical challenges. we pose our research questions as rq1) are zero-shot el models able to generalize effectively when introduced to a whole new kb, not included in their initial training? rq2) how much can zero-shot el models adapt to conversational settings without prior training? the contributions of this paper are: ## analysis scenarios to assess models based on practical constraints we perform the following groups of analysis. ## reddit conversational dataset for zero-shot el we introduce the reddit conversational el dataset, specifically curated for evaluating zero-shot el methods in conversational setup and with unseen kbs, used for our analysis scenarios. to curate this dataset we used the convokit's reddit corpus foot_1 @xcite , which includes subreddit posts and comments until october 2018, sourced from the broader pushshift reddit dataset foot_2 @xcite . convokit offers 948,169 subreddits, among which, we only opt for the discussions around each of the 16 domains used in zeshel @xcite (introduced in 4.2). we extract subreddits that contain a zeshel domain title in their name. from each reddit conversation, we extract its unique threads. in this context, a thread is a distinct path in a hierarchical structure of user utterances, beginning with an original post (the root) and encompassing all subsequent replies until the last reply (the leaf) @xcite . to create gold mention spans along with their gold fandom entities, we rely on instances where users include hyperlinks to the fandom website as a way of disambiguating their mention of an entity in their utterance. next, several preprocessing, pruning, and augmentation steps were performed: we checked the extracted annotations for instances where the gold mention and entity were exact matches. to avoid trivial disambiguation tasks, following @xcite , we ensured no more than 5% of our threads have such annotations. splitting the final data to train and test sets, we relied on conversation timestamps and annotation density (details in appendix c). dataset statistics can be found in ## experimental setup we detail the entity linking models, datasets and knowledge bases used in our experiments, as well as experimental details of our analysis setups. ## conclusions and future work this study re-examined the efficacy of current el models in conversational scenarios with limited data and kb resources. motivated by the realworld challenges frequent when integrating el components into conversational assistants, we recognized overlooked practical limitations in zeroshot el research. we showed that current zeroshot el models critically underperform when introduced to a new kb at inference time, due to shortcomings in both mention detection and entity disambiguation functions. these results highlight the need for designing better end-to-end zero-shot el systems that are reliable in various tasks and kb constraint scenarios. we conclude that the evaluation approaches being used so far in el literature to evaluate zero-shot el models are quite naive and not representative of the user's perspective on entity saliency, a crucial point when in interactive systems. for future work, we will leverage our curated dataset to advance model capabilities.