Typo-Robust Representation Learning for Dense Retrieval (2023)

Abstract: Dense retrieval is a basic building block of information retrieval applications. One of the main challenges of dense retrieval in real-world settings is the handling of queries containing misspelled words. A popular approach for handling misspelled queries is minimizing the representations discrepancy between misspelled queries and their pristine ones. Unlike the existing approaches, which only focus on the alignment between misspelled and pristine queries, our method also improves the contrast between each misspelled query and its surrounding queries. To assess the effectiveness of our proposed method, we compare it against the existing competitors using two benchmark datasets and two base encoders. Our method outperforms the competitors in all cases with misspelled queries. Our code and models are available at https://github.com/panuthept/DST-DenseRetrieval .

## introduction dense retrieval is a fundamental component in many information retrieval applications, such as open-domain question answering and ad-hoc retrieval. the objective is to score and rank a large collection of candidate passages based on their similarity to a given query. the performance of dense retrieval relies on representation learning. a popular approach is to finetune a pre-trained language model to create an embedding space that puts each query closer to its corresponding passages @xcite @xcite @xcite . one of the major challenges of dense retrieval is the handling of misspelled queries which induces representations of the misspelled queries to be closer to irrelevant passages than their corresponding passages. several studies have demonstrated that misspellings in search queries can substantially degrade retrieval performance @xcite , specifically when informative terms, such as entity mentions, are misspelled @xcite . to create a retrieval model that is capable of handling misspelled queries, researchers have proposed different training methods to align representations of misspelled queries with their pristine ones. @xcite devise augmentation methods to generate misspelled queries and propose training methods, typos-aware training and self-teaching (st), to encourage consistency between outputs of misspelled queries and their non-misspelled counterparts. alternatively, @xcite apply contrastive loss to enforce representations of misspelled queries to be closer to their corresponding non-misspelled queries. although these methods can improve the performance of retrieval models for misspelled queries, there is still a substantial performance drop for misspelled queries. in this paper, we propose a training method to improve dense retrieval for handling misspelled queries based on the following desired properties: • alignment: the method should be able to align queries with their corresponding passages. experimental studies were conducted to assess the efficiency of the proposed method in comparison to existing approaches. we conduct experiments based on two different pre-trained language models. we evaluate using two passage retrieval benchmark datasets, a standard one and a specialized one for misspellings robustness evaluation. for each dataset, we measure performance on both misspelled and non-misspelled queries, where the misspelled queries are both generated and realworld queries. the experimental results show that the proposed method outperforms the best existing methods for enhancing the robustness of dense retrieval against misspellings without sacrificing performance for non-misspelled queries. we summarize our contributions as follows: • we propose a novel training method to enhance the robustness of dense retrieval against misspellings by incorporating three desired properties: alignment, robustness, and contrast. ## methodology we propose a training pipeline to enhance the dense retrieval capability for handling spelling variations and mistakes in queries. as shown in figure 1 , the training pipeline comprises three steps. (i) query augmentation: we augment each query in the training set into multiple misspelled queries using the typo generators provided by @xcite . (ii) similarity score calculation: we compute similarity score distributions between queries and passages for passage retrieval and query retrieval tasks using in-batch negative queries and passages, with additional hard negative passages. ## conclusion this paper aims to address the misspelling problem in dense retrieval. we formulate three desired properties for making dense retrieval robust to misspellings: alignment, robustness, and contrast. unlike previous methods, which only focus on the alignment and robustness properties, our method considers all the desired properties. the empirical results show that our method performs best against misspelled queries, revealing the importance of the contrast property for handling misspellings.