Revisiting Query Variation Robustness of Transformer Models (2024)

Abstract: The most commonly used transformers for retrieval at present, BERT and T5, have been shown not to be robust to query variations such as typos or paraphrases. Although this is an important prerequisite for their practicality, this problem has hardly been investigated. More recent large language models (LLMs), including instruction-tuned LLMs, have not been analyzed yet, and only one study looks beyond typos. We close this gap by reproducing this study and extending it with a systematic analysis of more recent models, including Sentence-BERT, CharacterBERT, E5-Mistral, AnglE, and Ada v2. We further investigate if instruct-LLMs can be prompted for robustness. Our results are mixed in that the previously observed robustness issues for cross-encoders also apply to bi-encoders that use much larger LLMs, albeit to a lesser extent. While further LLM scaling may improve their embeddings, their cost-effective use for all but large deployments is limited. Training data that includes query variations allows LLMs to be fine-tuned for more robustness, but focusing on a single category of query variation may even degrade the effectiveness on others. Our code, results, and artifacts can be found at https://github.com/webis-de/EMNLP-24

## introduction despite their proficiency with natural language, transformer-based large language models (llms) trained for document ranking, like bert or t5, are not robust to ill-formed queries, including queries with typos and queries that omit less important words @xcite @xcite . @xcite reason that these variations are hardly represented in the llms' training data. however, query variations are the norm in practice: about 70% of information-seeking queries to web search engines are keyword queries @xcite instead of fully formed questions, and up to 26% of queries contain typos @xcite . however, due to their superior effectiveness, current information retrieval (ir) systems use these 'embedding models' foot_1 despite their lack of robustness. to our knowledge, this phenomenon has hardly been investigated with respect to information retrieval (section 2). @xcite contributed the most exhaustive study to date. they measure the changes in ranking effectiveness of various models for an originally intended, well-formed query from trec dl'19 @xcite and antique @xcite compared to randomly generated but semantically equivalent variations. however, no study yet investigated the ranking robustness of llm-based embedding models more recent than bert @xcite and t5 @xcite with respect to query variations. given the fast pace at which llms have been scaled, it is unclear whether scaling alone suffices to solve the problem. in this paper, we investigate the robustness of recent large language models against purely syntactic, query variants which preserve semantics. we reproduce penha et al.'s experiment on the robustness of cross encoders to different query variants from well-formed queries. in addition to comparing the ranking effectiveness of query variations to well-formed queries, we measure the anisotropy adjusted cosine similarity of the variations' embeddings compared to their respective original's to quantify the robustness of the embeddings, i.e., how much input variations affect the models' outputs. doing so, we also extend penha et al.'s study, to the state-of-the-art embedding models sentence-bert, characterbert, e5-mistral, angle, and ada v2. our reproduction results are consistent with previous work, showing that existing dense retrieval models are not robust to semantically identical illformed query variants. we also find llm-based embedding models to suffer from the same problem, but to a lesser extent. our experiments show that robustness to typos can be improved substantially using character-level transformers and typo aware pre-training, but this does not generalize to other query variations. we observe that focusing on a single category of query variation may even degrade effectiveness on other categories of variations. moreover, some models were less robust to keyword queries than to typos, presumably because transformer models use stop words to aggregate the context @xcite . ## related work though llms are effective at information retrieval and natural language processing tasks @xcite , they are not robust to language variations like typos or keywording @xcite @xcite . previous research identified two key contributing factors to this lack of robustness: (1) llms and ranking models are mostly trained on clean data, rendering variations like typos and keywords out of domain @xcite , and (2) character-level information is lost during tokenization @xcite . for example, 'weird' and its misspelling 'wierd' are tokenized by bert as weird and wi ##er ##d. typo awareness solutions to improve robustness of transformer-based models for information retrieval can be categorized into improving robustness using typo aware (pre-)training strategies @xcite @xcite , using datasets with input variations @xcite , and developing models using character-level tokenization schemes @xcite . @xcite propose pre-training using a typo aware masked language modeling strategy to strengthen a model's robustness to typos. previous studies improved typo robustness by finetuning models on datasets augmented with misspellings @xcite . @xcite train dual encoders using a contrastive loss to learn closer representations for words and their typos while keeping other words' representations apart. sidiropoulos and kanoulas find that typos in less prevalent words degrade effectiveness more than typos in common words and identify bert's word-piece tokenizer as a limiting factor in this respect. to address the tokenizer's shortcomings, zhuang and zuccon (2022) use characterbert @xcite , a bert variant that creates characterlevel token representations instead of word pieces. they fine-tune it on a typo-augmented dataset, minimizing two loss components: the kl-divergence of the relevance label distributions of a query and its typo variant, and contrastive cross entropy loss to retain effectiveness. @xcite call these components 'robustness' and 'effectiveness' and additionally propose 'alignment' to learn similar representations for queries and their variants. characterbert's character-level tokenizer is crucial for more typo robustness as it can better recognize small input variations. @xcite propose the lexical-aware attention module (lea), which adds a learnable bias to the attention scores based on text similarity. beyond typos the research mentioned above only focuses on typos. @xcite take the robustness analysis further by creating a test collection covering three more classes of query variations that retain semantics. to our knowledge, the only other test collection for query variations beyond typos is uqv100 @xcite . however, it does not guarantee identical semantics across query variations, nor does it specify the category of the variation. we make extensive use of penha et al.'s collection to study the query robustness of a spectrum of transformer-based retrieval models, ranging from the cross-encoders used to supplement the collection, to typo aware embedding models, to embedding models based on llms. robustness issues are also not confined to ir. ravichander et al. (2021), sidiropoulos et al. (2022), and qiang et al. (2024) observe a significant drop in effectiveness introduced by synonyms, paraphrasing, and errors induced by automatic speech recognition when providing a voice interface for language models. @xcite analyze the accuracy degradation due to typos, synonyms, repetition, and 'shortcuts' (providing part of the answer together with the prompt) observed with prompted embedding models on natural language tasks. by adding perturbed examples in fewshot prompts, they improved robustness to all variations except typos. ## methods and experimental setup this section describes the experimental setup consisting of the datasets with query variations, the embedding models used for the experiments and the experiments performed. ## results and discussion penha et al.'s experiments. then, we compare these results with various other models from the literature to see how they generalize. finally, we evaluate the impact that typo awareness can have by comparing the robustness of different architectures and fine-tuning cbert and prompt-tuning e5 using a training set created from the query variations of penha et al.'s dataset. 4.1 reproduction to reproduce the robustness results by penha et al. for bert and t5, we reran their code with slight modifications: foot_5 (1) we updated the versions in the requirements.txt since previous versions were not supported anymore, (2) we fixed minor runtime errors which presumably occurred due to the version updates, and (3) we resolved an error in the evaluation routine for bert on antique. as previously noted, penha et al. did not evaluate on antique's test set but on the validation split defined by ir_datasets (macavaney et al., 2021), antique/train/split200-valid, which is a split of antique's official training set, since antique officially does not have a validation set. however, penha et al. fine-tune bert on antique's official training set (antique/train) instead of ir_datasets' training set (antique/ train/split200-train), thus training on part of their test data. we also remap antique's graded relevance labels to the range 0-3 as described by hashemi et al. in its readme. foot_6 as we use the same test set, the same pre-trained t5 model and their code, we expect the results for t5 to be nearly identical to those reported by @xcite from what can reasonably be attributed to rounding differences during inference. for bert on trec dl'19 we expect similar results to the original paper, but discrepancies beyond rounding errors are expected due to the stochastic nature of fine-tuning. on antique, we expect that our results for bert are dramatically worse regarding each query variant with the largest expected drop for the original queries. ## conclusion we investigated the robustness of transformerbased retrieval models to query variations using a test collection with syntactic query variations that keep semantics. we reproduced penha et al.'s baseline results and extended them by assessing the robustness of (1) much larger models and (2) a model designed to be typo-robust. while we were not able to completely replicate the results by penha et al., we obtained similar results and reaffirmed their conclusions. we show that, while the typo-aware characterbert model was the most robust to typos, this did not lead to robustness to other types of query variations. finally, we observed that the largest model we tested, while dramatically less efficient than other models, was generally more robust or competitive with all the other tested models regarding all query variations. however, none of the models were robust to the query variations and all were the least robust to typos or keyword queries. our results indicate that focusing on typo robustness alone is not enough and highlight the need for datasets like penha et al.'s such that typos and keyword queries no longer are out-of-distribution for ir models.