Impacts of Misspelled Queries on Translation and Product Search (2024)

Abstract: Machine translation is used in e-commerce to translate second-language queries into the primary language of the store, to be matched by the search system against the product catalog. However, many queries contain spelling mistakes. We first present an analysis of the spelling-robustness of a population of MT systems, quantifying how spelling variations affect MT output, the list of returned products, and ultimately user behavior. We then present two sets of practical experiments illustrating how spelling-robustness may be specifically improved. For MT, reducing the number of BPE operations significantly improves spelling-robustness in six language pairs. In end-to-end e-commerce, the inclusion of a dedicated spelling correction model, and the augmentation of that model’s training data with language-relevant phenomena, each improve robustness and consistency of search results.

## introduction in many e-commerce settings that support users across multiple language backgrounds, machine translation (mt) is used to translate search queries from the user's preferred language into the primary language of the store in order to match those search queries against the product catalog. search queries, however, are prone to spelling mistakes (e.g., typos or misspelled words) or, more generally, spelling variations (e.g., leaving out diacritics as shorthand or due to keyboard limitations). robustness of mt systems in the face of such "noisy" input has been a persistent focus of study in both statistical and neural translation. however, most work has considered only mt performance in isolation: if standard mt metric scores and/or robustness scores increase, then the mt system is more robust and the improvement is successful. we consider by contrast in this paper the multilingual e-commerce setting: a specific application of mt where the system's output is not the end of the story. consider a product search system that delivers search results based on a user's input that has first undergone mt. the quality of the search results then partially depends on the mt system's robustness to misspellings or typographical preferences in the original input. an additional spelling correction step can also precede mt to further mitigate the impacts of noisy input. we illustrate that targeted improvements to the mt system and the spelling correction system can improve the robustness of the product search system as a whole to spelling variations. our contributions in this paper are as follows: using test sets of rightly and wrongly spelled search queries in multiple secondary languages, along with targeted metrics of mt robustness and search result difference (section 3), we quantify how the spelling-robustness of industrial-grade mt systems impacts the e-commerce experience. we show that mt can implicitly correct for spelling errors up to half the time, that the properties of the retrieval system also impact the search results, but that correctly spelled queries still tend to lead to better shopping outcomes (section 4). via modeling experiments, we demonstrate that mt's spelling-robustness can be significantly improved by reducing the number of bpe operations (section 5). we measure the effect of a dedicated spelling correction model, and we improve its contribution as well by augmenting its training data with language-relevant phenomena such as missing diacritics (section 6). ## related work our e-commerce scenario, where users' search queries are automatically translated to match the language of the product catalog, is fundamentally similar to the setups presented by @xcite and @xcite , or the qt alternative studied by @xcite . the most straightforward method of measuring mt robustness is via standard reference-based metric scores -such as bleu @xcite , chrf @xcite , or ter @xcite -on a test set of interest. in this formulation, robustness is judged by comparing the mt system's score when translating a damaged version of the input against its score on the corresponding "clean" text. the approach is popular for evaluating both smt and nmt performance in the face of spelling errors @xcite , social-media content @xcite , and non-parallel training data @xcite . we follow @xcite in dispreferring this approach. instead, researchers have proposed targeted robustness metrics that go beyond the standard scoring of an mt output against a reference translation @xcite @xcite . our analyses use several of the same metrics, though sometimes with minor differences to accommodate our particular scenario. see section 3.2 for details. prior work has demonstrated how mt robustness may be improved, either via data augmentation that targets different kinds of spelling mistakes or "noise" @xcite @xcite @xcite or via specific modeling improvements @xcite @xcite @xcite . the bpe-based case study we present in section 5 contributes to the modeling line of work. our work in section 6 meanwhile follows the data augmentation approach. of special note to section 5, @xcite tried adjusting the number of bpe operations, among other hyperparameters, in mt systems aimed at the translation of user-generated texts. their robustness application was therefore much more general than our present study of spelling mistakes, and it was evaluated from the point of view of overall mt performance only. our experiments are more narrowly focused, and our evaluation more targeted. ## improving mt robustness with bpe having seen how the innate spelling-robustness of mt systems can vary widely, we now examine how a system's robustness can be explicitly improved. intuitively, an mt system may struggle to translate a spelling mistake because the input looks unusual compared to the bulk of the model's training data. systems trained with byte-pair encoding (bpe) require their inputs to be broken down into a sequence of sub-word units that match the model's fixed vocabulary @xcite . a familiar input word should be more often directly contained in the model's bpe vocabulary; a rare (e.g., misspelled) word would need to be broken down into a relatively larger number of sub-words. consider the english search query samsung galaxy s21 and its misspelled counterpart samsung gslaxy s21. using the same bpe model, these two strings are encoded as follows: the wrongly spelled query requires six tokens to the rightly spelled query's four. we tested this intuition by building slates of mt system variants that differ in the number of bpe iterations ("operations") that were performed prior to model training. as the number of bpe operations -and thus the model's vocabulary -becomes smaller, rightly and wrongly spelled inputs should be expressed in more and more similar ways. in the limit, when the model vocabulary consists of single characters only, there should be no difference in the nature of sub-words needed to express the two different types of query, and almost no difference in the pre-processed query lengths. ## improving search robustness with spelling correction while we can directly improve the robustness of an mt system to misspelled queries, in an ecommerce system, it is also possible to include an explicit spelling correction step in order to mitigate downstream impacts. we now explore the extent to which a spelling correction model can retain the performance of mt and search systems. ## conclusions this work began by illustrating the surprising range of spelling-robustness in a population of mt systems. we demonstrated the utility of three targeted metrics for quantifying robustness, each offering a somewhat different view of how mt output is disrupted by the presence of spelling variations in the input. we also related the disruption in mt to the disruption in search results and user behavior in a cross-lingual e-commerce setting. we then showed through practical experiments how reducing the number of bpe operations during mt training significantly improves spellingrobustness across five out of six language pairs while having a less systematic effect on overall translation quality. a second set of experiments demonstrated how a dedicated spelling correction model improves search robustness, furthered by a targeted improvement to the model through data augmentation.