Dealing with Typos for BERT -based Passage Retrieval and Ranking (2021)

Abstract: Passage retrieval and ranking is a key task in open-domain question answering and information retrieval. Current effective approaches mostly rely on pre-trained deep language model-based retrievers and rankers. These methods have been shown to effectively model the semantic matching between queries and passages, also in presence of keyword mismatch, i.e. passages that are relevant to a query but do not contain important query keywords. In this paper we consider the Dense Retriever (DR), a passage retrieval method, and the BERT re-ranker, a popular passage re-ranking method. In this context, we formally investigate how these models respond and adapt to a specific type of keyword mismatch – that caused by keyword typos occurring in queries. Through empirical investigation, we find that typos can lead to a significant drop in retrieval and ranking effectiveness. We then propose a simple typos-aware training framework for DR and BERT re-ranker to address this issue. Our experimental results on the MS MARCO passage ranking dataset show that, with our proposed typos-aware training, DR and BERT re-ranker can become robust to typos in queries, resulting in significantly improved effectiveness compared to models trained without appropriately accounting for typos.

## introduction passage ranking is a core task for many information retrieval related applications. in the context of conversational search and question answering, for example, passage ranking is often the first step in the system's pipeline: thus the quality of the ranking results will affect the effectivenesses of the downstream tasks. traditional passage ranking models, such tf-idf and bm25, use exact keyword matching signals, where a retrieved passage must contain at least one of the query's keywords. this mechanism however limits the capability of these models to retrieve passages that are semantically relevant but use different keywords: this is the well-known vocabulary mismatch problem. recent advances in nlp have seen the introduction of deep language models @xcite @xcite ; bert @xcite in particular has shown generalised promise in language understanding tasks. bert adopts the transformer encoder @xcite as model architecture and uses wordpiece token embeddings @xcite as model inputs. this design allows bert to deal with the vocabulary mismatch problem. hence practitioners have turned to design bert-based passage ranking models @xcite . two main directions have been adopted to exploit bert for effective passage ranking: the input to bert is @xmath0, @xmath1, or the query likelihood estimated by the bert model (zhuang and zuccon, 2021b). a key drawback of bert re-ranker models is that multiple inferences are required at query time: this is a computationally expensive process which results in high query latency @xcite @xcite . thus the use of these methods is confined to second stage re-ranking. in principle, these methods are not affected by keyword mismatch because they use the latent embedding space to estimate the relevance of a query to a passage. this is supported by recent work that has shown the dr and bert re-ranker provide better semantic matching @xcite @xcite . in this paper we investigate the impact of a specific type of keyword mismatch: that caused by the presence of typos in the query. traditional exact keyword matching methods perform badly on queries that contain typos. extra query processing steps, such as spelling correction, are required for these methods to be tolerant to typos in queries @xcite . on the other hand, it is expected that bert-based models can handle typos occurring in queries well. this is because bert uses the wordpiece algorithm which splits a keyword that does not match an entry in the bert vocabulary (typos are likely to not be present in this vocabulary) into character-level sub-tokens. this can be used to produce embeddings for out-of-vocabulary keywords, which are then passed as input to the bert encoder. however, this intuition has never been tested before, and the capability of bertbased passage ranking models to deal with typos in queries has not been quantified. to address this gap, we first formally investigate how the bert-based dr and re-ranker respond and adapt to queries that contain typos. specifically, we use different typo generators to produce typos for queries; we then compare the effectiveness of the rankers when using queries with typos vs. without typos. interestingly, we find that these models fail to handle queries with typos -typos can lead to a significant drop in effectiveness for both dr and bert re-ranker. in order to solve this issue and obtain typo-robust ranking models, we then propose a simple typos-aware training strategy, in which queries with typos are produced and used also for training. our experimental results on the ms marco passage ranking dataset show that, with our typosaware training, dr and bert re-ranker can become robust to typos in queries, without loss in effectiveness for queries without typos. ## methodology with respect to bert-based models for passage retrieval and typos in queries, we investigate the following research questions: • rq1: what is the impact of typos in queries on bert-based dr and re-ranker effectiveness? • rq2: do different typo types affect the effectiveness of the bert-based methods differently? • rq3: does the proposed typos-aware training improve the effectiveness of the bert-based methods on queries with typos? does it hurt their effectiveness on queries without typos? ## results empirical results are reported in in terms of the impact of different types of typos (rq2), the results show that different typos have similar impact: they all hurt effectiveness heavily. dr appears however more tolerant to randdelete typos (with a ≈ 5% smaller loss in mrr@10 than for other types of typos), while bert re-ranker losses are generally uniform across typos types. to answer rq3, we compare the results of models produced with typos-aware training vs. with the standard training. despite the typos-aware training, both dr and bert re-ranker display significant losses in effectiveness when dealing with typos queries, compared to the original queries ( figure 1 presents the rank loss obtained by dr when answering typos queries in place of the original queries (plot averaged across all types of typos; individual typos types show similar trends). a negative loss of n means when using typos queries, the first relevant document is retrieved n rank positions after that obtained when using the original queries. the figure shows that typos-aware training consistently provides smaller losses than the standard training. we also note there are few cases (≈ 300 queries) in which typos queries provide gains compared to the original query. queries with large losses often have typos for keywords that are essential to determine the intent of the query. typos queries that exhibit gains generally display typos on non-essential keywords, e.g., stopwords. ## a case study the results presented in the previous sections are conducted with synthetically generated typo queries. accurate analysis of the ms marco dataset revels the presence of a very limit number of queries containing typos -these typos are legitimate errors made by the user issuing the query. for instance, the ms marco dev set contains the mistyped query -"sydeny climate" foot_1 (qid: 506025). without typos-aware training, the considered dr cannot retrieve the relevant passage in the top 1,000 results. however, with typos-aware training, the dr is able to rank the relevant passage at rank 127 for this typo query. this suggests that drs trained with our typos-aware training with synthetic typo generation may be able to generalize to real-world typo queries, aside from those synthetic (though realistic) typo queries we considered in our extensive empirical evaluation. in future work, we want to further test our proposed typos-aware training with more real-world typo queries by acquiring a real query log with typos and perform relevance annotations on the ms marco passage collection. ## conclusion in this paper we studied the impact of typos in queries on popular bert-based passage retrieval methods. we reported significant drops in effectiveness across different types of typos for both dr and bert re-ranker: these methods are not tolerant to typos in queries when solely relying on the bert encoder. we then proposed a typos-aware training strategy for dr and bert re-ranker, which controls the exposure of the models to queries with typos during training. with our typos-aware training, both dr and bert re-ranker showed to be much more tolerant to typos in queries. we believe our typos-aware training can be used (more extensively than in this paper) as a standard data augmentation step in the dr and bert re-ranker's training loop since the computations for typos generation are very light and can provide extra gains on typos queries, without hurting effectiveness on queries without typos. code, typos queries and results files at https://github.com/ielab/ typos-aware-bert .